2017 · Transformers
Attention Is All You Need
The transformer replaces recurrence with attention. It became a foundation for language models and many other systems that process sequences.
Paper & contextAshish Vaswani
Language models
Using attention to process sequences in parallel.
Vaswani studies architectures for learning from sequences and other structured data. The transformer is the starting point for this selection, followed by research on attention, efficient training, visual models, and large-scale pretraining data.
Selected work
10 papers
2020 · Language models
Efficient Content-Based Sparse Attention with Routing Transformers
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2019 · Language models
Stand-Alone Self-Attention in Vision Models
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2026 · Language models
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2025 · Language models
Essential-Web v1.0: 24T tokens of organized web data
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2025 · Language models
Practical Efficiency of Muon for Pretraining
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2025 · Language models
Rethinking Reflection in Pre-Training
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2021 · Language models
The Efficiency Misnomer
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2021 · Language models
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2021 · Language models
Simple and Efficient ways to Improve REALM
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & contextAn independent editorial profile. Inclusion does not imply Council membership or endorsement. Research is collaborative; coauthorship does not imply sole credit.
Selection & sources