A bigger model can be the wrong purchase.
A training budget can buy more parameters, more exposure to data, or a mixture. Hoffmann and colleagues used over 400 smaller training runs to investigate that allocation. Their recipe increased model size and training tokens roughly together as compute grew.
The experiment that made the point
Chinchilla used 70 billion parameters and 1.4 trillion training tokens. With approximately Gopher’s training compute, it outperformed the 280-billion-parameter model on the reported evaluations. The smaller network received far more training.
What the curve below means
This graphic uses the paper’s separate parametric fit. Its mathematical minimum can differ from the actual Chinchilla configuration. Each point divides one compute budget between parameters and tokens; the curve estimates the resulting loss.
Find the balance between model and data.
Move the model-size slider. With a fixed training budget, a larger model gets fewer training tokens.
Assumptions: a historical Chinchilla loss fit and C ≈ 6ND. Data quality and architecture stay fixed; lower loss does not measure intelligence.
At this budget, the equation’s minimum is around 32.5B parameters and 3.02T tokens.
The equation and assumptions
Appendix D.2 fits L = 1.69 + 406.4/N⁰·³⁴ + 410.7/D⁰·²⁸, using raw parameter and token counts. We approximate training compute as C = 6ND; the reference budget is 5.88 × 10²³ FLOPs. This is one fit from the paper, not a universal optimal recipe. Loss measures prediction on the study’s text distribution, not intelligence. Architecture, data quality, inference expense, and modern training changes are outside this illustration.
What the result does not establish
The large-scale comparison involved two main runs, and the allocation analysis used less than one pass over the data. Extrapolating much further, repeating data, or changing architecture can change the recipe.
Original sources: Kaplan et al. (2020) and Hoffmann et al. (2022), including Appendix D.2. Continue with the scaling-laws reading guide.
Source: Training Compute-Optimal Large Language Models. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.