Scaling turns a training run into a planning problem.
Kaplan and colleagues measured next-token prediction loss while varying model size, data, and training computation. Across the tested ranges, loss followed smooth power-law trends: a proportional increase in resources produced a predictable, diminishing improvement in that measurement.
Why that changes how a lab trains models
Instead of choosing a model size by intuition, a lab can run smaller experiments, fit a curve, and use it to plan a larger run. In this study, the compute-efficient allocation favored large models trained relatively briefly, stopping before convergence.
The next question: how much data?
Chinchilla revisited the allocation in 2022. The interactive curve below uses that later paper’s fit, so it is a comparison with the next step in the research—not a reproduction of Kaplan’s coefficients.
Find the balance between model and data.
Move the model-size slider. With a fixed training budget, a larger model gets fewer training tokens.
Assumptions: a historical Chinchilla loss fit and C ≈ 6ND. Data quality and architecture stay fixed; lower loss does not measure intelligence.
At this budget, the equation’s minimum is around 32.5B parameters and 3.02T tokens.
The equation and assumptions
Appendix D.2 fits L = 1.69 + 406.4/N⁰·³⁴ + 410.7/D⁰·²⁸, using raw parameter and token counts. We approximate training compute as C = 6ND; the reference budget is 5.88 × 10²³ FLOPs. This is one fit from the paper, not a universal optimal recipe. Loss measures prediction on the study’s text distribution, not intelligence. Architecture, data quality, inference expense, and modern training changes are outside this illustration.
What the result does not establish
Loss on held-out text is narrower than reasoning ability, factual accuracy, or usefulness. The fitted trends depend on the data and training setup; an extrapolation beyond the experiments remains a hypothesis.
Original sources: Kaplan et al. (2020) and Hoffmann et al. (2022), including Appendix D.2. Continue with the scaling-laws reading guide.
Source: Scaling Laws for Neural Language Models. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.