StateBench reveals limits in long-context state tracking
New benchmark tests show transformers struggle with complex state tracking while Gated DeltaNet models generalize effectively.
StateBench is a new benchmark designed to evaluate how language models track evolving entity information in long documents. It defines four difficulty levels, ranging from immutable assignments to complex reference-based updates. The hardest level involves tracking successive swaps of five objects. Theoretical analysis suggests transformers have inherent limits on these hard variants, particularly those with long dependency chains.
Testing of open pretrained models, including Olmo 3 and smaller variants, showed strong performance on simple immutable tasks. However, accuracy declined on aperiodic problems as inputs grew, and models remained at chance levels on general finite problems. Frontier models such as Gemini 3.8 Flash and Claude Opus 5.5 also saw accuracy drop sharply on longer inputs, falling to 13% or lower on the most difficult sequences.
In contrast, Gated DeltaNet variants with flexible memory updates generalized almost perfectly across easier levels. One variant maintained near-perfect accuracy on the hardest level, extending to roughly ten times the length of its training sequences. This highlights a significant performance gap between standard transformer architectures and newer memory-based models in long-context state tracking.
Sources
Every sentence of this story was checked against facts quoted from these articles. How we make the news
Spotted an error? Tell us
