2017 · Transformers
Attention Is All You Need
The transformer replaces recurrence with attention. It became a foundation for language models and many other systems that process sequences.
Paper & contextNoam Shazeer
Language models
Combining attention and sparse conditional computation.
Shazeer’s work connects attention-based architectures with sparse conditional computation. This selection includes the transformer and mixture-of-experts research, following approaches that increase model capacity while controlling the computation used for each input.
Selected work
10 papers
2017 · Architecture
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Sparsely gated mixture-of-experts layers explore conditional computation using a learned routing network.
Paper & context2021 · Architecture
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Switch Transformers studies sparse mixture-of-experts models, activating selected expert networks rather than every parameter.
Paper & context2022 · Efficient AI
PaLM: Scaling Language Modeling with Pathways
Selected research in efficient ai, language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2025 · AI for science
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Selected research in ai for science, reinforcement learning. Read the full paper, including the methods, experiments, and reported results.
Paper & context2025 · AI for science
Gemma 3 Technical Report
Selected research in ai for science, efficient ai. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Language models
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Efficient AI
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Selected research in efficient ai, language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Language models
LaMDA: Language Models for Dialog Applications
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & context2021 · Language models
Primer: Searching for Efficient Transformers for Language Modeling
Selected research in language models. Read the full paper, including the methods, experiments, and reported results.
Paper & contextAn independent editorial profile. Inclusion does not imply Council membership or endorsement. Research is collaborative; coauthorship does not imply sole credit.
Selection & sources