SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

The idea in plain language.

Move quantization difficulty into weights before running the lower-precision model.

How it works

Activation outliers can make low-bit arithmetic inaccurate. SmoothQuant applies an equivalent transformation that rescales activations and compensates in the weights. This reduces the difficult activation ranges without changing the unquantized operation. The transformed model is then quantized for lower-precision inference, aiming to reduce memory use and improve hardware efficiency without additional model training.

What to keep in mind

Calibration data and hardware support matter. Accuracy and speed must be measured on the actual model, workload, and serving implementation.

Source: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.