The idea in plain language.
Most values can use fewer bits if difficult outliers receive special treatment.
How it works
Quantizing every feature in the same way can lose information carried by outlier dimensions. LLM.int8() uses an 8-bit computation for most values and separates exceptional feature dimensions into a higher-precision calculation. This mixed approach targets inference memory savings while preserving model behavior on the paper's evaluated models and tasks.
What to keep in mind
Memory savings do not directly imply the same speedup on every device. Hardware support, model structure, and outlier behavior affect practical results.
Source: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.