Transformers · ORIGINAL RESEARCH

Attention Is All You Need

The transformer replaces recurrence with attention. It became a foundation for language models and many other systems that process sequences.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones and colleaguesSource date: 2017-06-12

Why use attention instead of reading a sequence step by step?

The Transformer makes relationships between tokens directly available to the model and lets more training work happen in parallel.

Baseline, result & limitation

Baseline / setup: Recurrent and convolutional translation systems; the paper compares translation quality and estimated training compute.
Reported result: The large Transformer reports 28.4 BLEU on WMT 2014 English-to-German translation, exceeding the prior results listed in Table 2.
Keep in mind: These translation experiments do not establish general reasoning ability. Standard self-attention also has a quadratic sequence-length cost.

Before you read A token is a piece of text; an attention weight controls how much one position uses information from another.

Artificial Post reading note · reviewed September 12, 2026

Opening the original paper…

Opening the paper…

KEEP FOLLOWING THE IDEA