Declarative Attention protocol lets language models control their own context search
By allowing LLMs to restrict their attention to global, focused, or local regions, the framework reduces attended tokens during decoding by up to 52%.
Original cover art, generated for this story. THE VISSION does not republish third-party press imagery.
- A research preprint published on arXiv on September 2 introduces Declarative Attention (DA), a protocol that lets LLMs decide which context segments to attend to.
- Evaluating the protocol on Gemma-4-31B and Qwen-3.6-27B demonstrated token reduction of 52.0% and 31.1% respectively during chain-of-thought decoding.
- The token reduction incurs only a modest accuracy drop of 1.27 to 2.75 percentage points across 15 long-context tasks, with the gap narrowing on larger models.
A research preprint published on arXiv on September 2 has introduced Declarative Attention (DA), a protocol that allows large language models (LLMs) to dynamically control their own attention layers during inference. Developed to address the severe computational overhead of scanning massive key-value (KV) caches, the framework shifts attention management from fixed mathematical heuristics to the model's own reasoning process.
In long-context conversations spanning over a million tokens, LLMs typically read the entire KV cache to generate each new token, despite focusing on only a tiny fraction of the context. DA solves this by partitioning the model's generation into three distinct modes: global, focus, and local. During chain-of-thought reasoning, the model explicitly 'declares' when it needs to scan the full context (<global>), restrict attention to a specific identified text block (<focus>), or limit attention solely to its most recent outputs (<local>).
The researchers evaluated the DA protocol across 15 long-context tasks using off-the-shelf models, including Gemma-4-31B and Qwen-3.6-27B. Gemma-4-31B achieved a 52.0% reduction in total attended tokens during decoding, while Qwen-3.6-27B saw a 31.1% reduction. This substantial drop in computational overhead incurred only a minor performance cost, with accuracy dropping by a modest 1.27 to 2.75 percentage points across evaluations. The authors noted that this performance gap shrivels as model scale increases, suggesting that larger models are better at managing their own attention budgets.
Scanning the entire KV cache for every single token in long-context inference is a major cost bottleneck for multi-agent systems and document processing. Declarative Attention proves that LLMs are capable of acting as their own attention governors, selectively ignoring massive swaths of irrelevant context during multi-step reasoning. This self-directed sparse attention could drastically cut the API and compute costs of long-context applications, making long-context model deployment significantly more sustainable and scalable.