Editorial selection · September 12, 2026

Five papers worth understanding

A reading path through model architecture, training and retrieval. Selected from 2017–2024 research.

5 of 5 papers

Architecture · 2017
Attention Is All You Need
The Transformer makes relationships between tokens directly available to the model and lets more training work happen in parallel.
Baseline, result & limitation for Attention Is All You Need
Baseline / setup
Recurrent and convolutional translation systems; the paper compares translation quality and estimated training compute.
Reported result
The large Transformer reports 28.4 BLEU on WMT 2014 English-to-German translation, exceeding the prior results listed in Table 2.
Keep in mind
These translation experiments do not establish general reasoning ability. Standard self-attention also has a quadratic sequence-length cost.
Before you read A token is a piece of text; an attention weight controls how much one position uses information from another.
Check the original manuscriptSection 6.1 and Table 2: translation results ↗Section 3: architecture; Section 4: attention costs ↗
Read the paper here : Attention Is All You Need
Scaling · 2020
Scaling Laws for Neural Language Models
A fitted trend can turn model size, data and computation into a planning problem instead of three unrelated guesses.
Baseline, result & limitation for Scaling Laws for Neural Language Models
Baseline / setup
The authors vary Transformer size, dataset size and training compute, measuring prediction loss on held-out text.
Reported result
Loss follows approximate power laws over the tested ranges. Their compute-efficient allocation favors larger models trained short of convergence.
Keep in mind
Prediction loss is not intelligence. The fits depend on the training setup, and Appendix C cautions against extrapolation outside the experiments.
Before you read Loss measures prediction error. A power law describes how that measurement changes when a resource is multiplied.
Check the original manuscriptSection 6: allocating the compute budget ↗Appendix C: caveats ↗
Read the paper here : Scaling Laws for Neural Language Models
Scaling · 2022
Training Compute-Optimal Large Language Models
Chinchilla revisits the balance between parameters and training tokens. Buying more model capacity can leave too little computation for training it.
Baseline, result & limitation for Training Compute-Optimal Large Language Models
Baseline / setup
Gopher: 280 billion parameters. Chinchilla uses 70 billion parameters and 1.4 trillion tokens at approximately the same training compute.
Reported result
Chinchilla reports 67.6% average five-shot MMLU accuracy versus Gopher’s 60.0%: a 7.6 percentage-point difference.
Keep in mind
Only two comparable large runs anchor the scale-up test; optimizer, tokenizer and data-mixture details also differ. The scaling analysis uses less than one data epoch.
Before you read Parameters are learned weights; training tokens measure how much text the model processes during training.
Check the original manuscriptSection 4.2.2 and Table 6: the MMLU comparison ↗Sections 4.1 and 5: setup differences and limitations ↗
Read the paper here : Training Compute-Optimal Large Language Models
Retrieval · 2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
RAG combines generated answers with retrieved documents, giving a model an external source of knowledge that can be inspected and updated.
Baseline, result & limitation for Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Baseline / setup
Closed-book language models and retrieval-based question-answering systems. RAG combines a DPR retriever with a BART generator.
Reported result
On Natural Questions, RAG-Sequence reports 44.5 exact match, compared with 41.5 for DPR in Table 1.
Keep in mind
The experiment uses a Wikipedia index and task-specific training. Better benchmark answers do not guarantee that every generated claim is supported by a retrieved passage.
Before you read Retrieval selects passages for a question; generation writes an answer using the question and those passages.
Check the original manuscriptSection 4.1 and Table 1: question-answering results ↗Section 2: retriever and generator ↗
Read the paper here : Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Training cost · 2024
DeepSeek-V3 Technical Report
This report connects sparse model design, numerical precision and systems engineering to a concrete training-compute account.
Baseline, result & limitation for DeepSeek-V3 Technical Report
Baseline / setup
The architecture builds on DeepSeek-V2. Table 1 accounts separately for pretraining, context extension and post-training.
Reported result
The authors report 2.788 million H800 GPU-hours. At their assumed $2 per GPU-hour, the rental equivalent is $5.576 million.
Keep in mind
That figure excludes prior research and ablations. It is neither the company’s total development spending nor a current cloud quote; hardware and reporting scopes matter.
Before you read A GPU-hour is one GPU used for one hour. More GPUs can shorten a run without reducing its total GPU-hours.
Check the original manuscriptIntroduction, Table 1: training costs and exclusions ↗Sections 2 and 3: architecture and infrastructure ↗
Read the paper here : DeepSeek-V3 Technical Report

Opening five good reads…

Browse the wider indexed edition

Automated collections retain original publication dates. These are separate from the editorial selection above.

THE ARTIFICIAL POST / THE WEEKLY EDITION

A world worth
understanding.

The opening collection

Follow the RSS feedRead free on the web. Email delivery is not active yet.

THE OPENING EDITION

A few good places to begin.

An introductory collection from across the AI world. Historical items retain their original publication dates.

A world worthunderstanding.

A few good places to begin.

A world worth
understanding.