SpecimenVOL. I
Import a corpus
The shelfSPECIMEN / REV-ATTENTION
LITERATURE REVIEW · 12 PAPERS

Attention mechanisms in neural sequence models

A survey of architectural innovations, 2014 – 2024

Maya AlvesPhD Candidate · ML SystemsUC Berkeley EECS2026-04-19
Abstract

Attention mechanisms reshaped sequence modelling between 2014 and 2024, progressing from additive soft attention to the scaled dot-product transformer and its sparse / linear descendants. This review clusters 12 canonical works into three strands (foundations, efficiency, and interpretability), identifies one under-cited bridge between the efficiency and interpretability literatures, and notes the shift in benchmarking conventions that clouds cross-strand comparison.

§ I — Citation graph
hover a theme to tint:
§ II — Synthesis

Foundations established attention as a first-class operator.

Bahdanau et al. introduced additive attention as a soft alignment mechanism for encoder-decoder NMT [bahdanau 14], directly motivating the fully-attentional Transformer [al. 17]. Segment-level recurrence and relative position encodings extended the base architecture to longer sequences [al. 19], closing the foundations strand by 2019.

Efficiency work produced sub-quadratic variants.

By 2019 the quadratic cost of self-attention had become the dominant constraint. Sparse attention reduced cost to O(n√n) via structured patterns [al. 19]; local-plus-global windowing produced Longformer [beltagy 20]; low-rank projection and kernel approximation yielded linear-time variants [al. 20] [al. 20].

Interpretability remained cautious.

Empirical probes found dependency-like structure in individual heads [al. 19] and a sparse set of specialist heads carrying most of the function [al. 19]. A contemporaneous adversarial result showed attention is a weak explanatory signal in the sense of Lipshitz-stable maps [jain 19].

Bridging work is rare and recent.

Only two works meaningfully bridge efficiency and interpretability: a circuit-level algebraic decomposition [al. 21] and the identification of induction heads as a mechanistic basis for in-context learning [al. 22]. The latter has not yet been reproduced on any sparse variant.

§ III — Gap candidates
  1. 01Efficiency × Interpretability under-cited

    Only two papers (Elhage 2021, Olsson 2022) bridge the efficiency and interpretability strands. The induction-head mechanism has not been analyzed in any sparse-attention variant to date. High-leverage gap for a PhD proposal.

  2. 02Benchmark drift

    Foundations-era papers benchmark on WMT-14; efficiency papers prefer PG-19 and arXiv-summarization; interpretability papers use SQuAD probes. No single paper reports on all three — comparative claims are structurally weak.

§ IV — Corpus · 12 papers
  1. [bahdanau2014]Bahdanau, Cho, Bengio (2014). Neural machine translation by jointly learning to align and translate. ICLR. 1409.0473
  2. [vaswani2017]Vaswani et al. (2017). Attention is all you need. NeurIPS. 1706.03762
  3. [dai2019]Dai et al. (2019). Transformer-XL: long-range language modeling. ACL. 1901.02860
  4. [child2019]Child et al. (2019). Generating long sequences with sparse transformers. arXiv. 1904.10509
  5. [beltagy2020]Beltagy, Peters, Cohan (2020). Longformer: the long-document transformer. arXiv. 2004.05150
  6. [wang2020]Wang et al. (2020). Linformer: self-attention with linear complexity. arXiv. 2006.04768
  7. [choromanski2020]Choromanski et al. (2020). Rethinking attention with performers. ICLR. 2009.14794
  8. [clark2019]Clark et al. (2019). What does BERT look at? An analysis of attention. BlackboxNLP. 1906.04341
  9. [voita2019]Voita et al. (2019). Analyzing multi-head self-attention: specialized heads do the heavy lifting. ACL. 1905.09418
  10. [jain2019]Jain, Wallace (2019). Attention is not explanation. NAACL. 1902.10186
  11. [elhage2021]Elhage et al. (2021). A mathematical framework for transformer circuits. Anthropic. anthropic.com/circuits
  12. [olsson2022]Olsson et al. (2022). In-context learning and induction heads. Anthropic. arxiv.org/2209.11895