Scaling laws.
Bigger is only part of it.

What is an AI scaling law?

A scaling law is an empirical relationship: measure a system at several sizes, fit a mathematical pattern, and test whether that pattern predicts what happens at a larger scale. It is useful for planning experiments, and it can be revised when experiments reveal a different relationship.

For language models, a common measurement is loss: how surprised the model is by the next token in held-out text. Lower loss means better prediction on that distribution. It does not, by itself, measure whether a system is trustworthy or useful for your task.

Kaplan and colleagues: performance becomes more predictable.

The 2020 paper studied language-model loss as model size, dataset size, and training computation changed. Across the tested ranges, the authors found smooth power-law trends. Those fits suggested a way to forecast a larger experiment from smaller ones.

Under their experimental setup, the compute-efficient strategy favored very large models trained on a comparatively modest amount of data, stopping before convergence. The result was a resource-allocation recipe with assumptions, rather than a promise that any larger model will be better at everything. Kaplan et al., 2020 ↗

Chinchilla: use the training budget differently.

In 2022, Hoffmann and colleagues trained hundreds of models to revisit the allocation between parameters and training tokens. They found that the compute-optimal allocation in their experiments scaled the two approximately together.

Their 70-billion-parameter Chinchilla model used about the same training compute as the 280-billion-parameter Gopher while processing four times as much data. It performed better on the evaluations they reported. That changed the practical question from “How large can we make the model?” to “How much data should this model see?” Hoffmann et al., 2022 ↗

There is no contradiction in a newer study changing the recipe. The fitted relationship depends on the experiments, training choices, and range being measured.

Test-time scaling: spend more effort on the answer.

A trained model can use additional computation by producing candidate solutions, revising an answer, or searching with a verifier. Snell and colleagues studied how to allocate that effort on reasoning tasks. The best strategy depended on the difficulty of the problem and the model’s ability to make progress on it.

Their findings make test-time compute a separate design choice. They do not mean that unlimited retries can solve every problem, or that the same strategy transfers unchanged across tasks. Snell et al., 2024 ↗

How to read the next “scaling” claim.

1. Find the measured outcome. Is it prediction loss, benchmark accuracy, a human preference, or successful completion of a real task?
2. Find what was held fixed. The model, data, training budget, and answer-time budget can each change the comparison.
3. Check the tested range. A line fitted to small experiments is a hypothesis about larger ones until it is tested there.
4. Look for the cost that was left out. Training once and serving many answers create different resource constraints.

Three papers. One useful reading order.