Declarative Attention: reducing KV-cache reads

A preprint by Namgyu Ho and colleagues, posted to arXiv on September 2, lets a model declare global, focus and local context regions to reduce KV-cache reads.

The authors report 31.1-52.0 per cent fewer attended tokens with modest accuracy loss.

Testing covered two model families; there is no production validation.