The idea in plain language.
Optimizer history can use lower precision when quantization adapts to local value ranges.
How it works
Optimizers such as Adam store statistics for each parameter. This paper divides those state tensors into blocks and quantizes each block separately, reducing the effect of large values on nearby small ones. Additional quantization and embedding choices aim to preserve stable training. The resulting memory savings can make room for model parameters or larger training configurations.
What to keep in mind
This reduces optimizer-state memory rather than every component of training memory. Compatibility and stability still require evaluation for the chosen model and optimizer.
Source: 8-bit Optimizers via Block-wise Quantization. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.