#long-context

Sparse Attention

A transformer efficiency technique where each token attends only to a relevant subset of other tokens, enabling near-linear scaling with context length.