Every long-document task hides a state-tracking problem. Reading code means tracking variables as they are assigned and reassigned; following a story means tracking who holds what. The failure mode is rarely the prose — it is the bookkeeping, and it gets harder as the document gets longer.
StateBench, released on 3 November by the Allen Institute for AI with a tech report, dataset and code on Hugging Face, measures that bookkeeping directly. Its design choice is the interesting part: instead of testing the hardest cases only, it decomposes state tracking into four levels of increasing difficulty, framed as four kinds of variable assignment.
Four Levels
- R-trivial — immutable assignment (
x = 4). The model only recovers the first value; like a birth date, it never changes. - Aperiodic — reassignment by value (
x = 4thenx = 2). Only the latest value matters; a thermostat. - Soluble — self-referential updates (
x = x + 5). Changes must be carried forward, not looked up; a running game score. - General finite — assignment by reference (
x = y, swapping two variables). The classic S₅ problem: a ball hidden under five cups, with the cups repeatedly swapped.
The hierarchy is not arbitrary — it comes from Krohn–Rhodes theory, which decomposes state-tracking problems into atomic primitives. All four benchmark variants use the same number of variables and place assignments at the same points in the sequence; only the operation changes.
What It Found
On pretrained open models — Olmo 3 (transformer) versus Olmo Hybrid (adds Gated DeltaNet layers), plus 275M and 2.7B variants, none trained beyond 64,000 tokens — nearly all scored well on R-trivial, degraded on Aperiodic as inputs grew, and sat at chance on General finite. Format mattered: a 275M hybrid was near ceiling on RULER-style R-trivial problems but near chance on the same problems framed as spreadsheet edits.
On frontier models — Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna — the same pattern, sharper. All four handled R-trivial and Aperiodic well, then fell off a cliff on Soluble and General finite as inputs grew, dropping to 13% or less on the longest General finite problems.
The decisive experiment was direct training. Training 275M models from scratch on up to 256 assignments and testing out to 64,000 assignments — 250x the training length — separated architecture from data. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, with the flexible-memory Gated DeltaNet staying near-perfect out to roughly 10x its training length. Transformers learned the easier levels and degraded on longer ones, and never learned General finite at all. That difference is architectural, not a data problem.
Why It Matters
For anyone building agents that carry state over long contexts, this is a selection criterion rather than a benchmark curiosity: attention-only designs fail on exactly the dependency chains that long workflows generate, and a running recurrent memory measurably survives them. Open questions remain — the placement of limited-memory Gated DeltaNet in the hierarchy, numerical precision effects, and the fact that StateBench assumes a fixed variable count, while real conversations keep introducing new entities.
Blog: StateBench on Hugging Face · Tech report · Dataset