Study guides · Intermediate

Quantization error and inference performance

Separate quantization scale, rounding, and clipping, then examine outliers, layer-specific sensitivity, and actual execution formats.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define the quantization grid
  2. Separate rounding and clipping
  3. Inspect outlier-driven scaling
  4. Compare scale granularity
  5. Evaluate more than reconstruction
  6. Do not infer speed from bit width alone
  7. Ask Logeia about this articleCheck your understanding

Define the quantization grid

Quantization maps higher-precision values onto a limited set. A simple affine rule rounds x divided by a positive scale s, adds zero point z, and clips to an integer range; reconstruction multiplies q minus z by s. The stored integer is not the original real value. Scale, zero point, integer limits, and rounding convention are all needed to interpret it. This article uses a uniform grid to isolate the mechanism rather than implying that every deployed quantizer has the same levels or storage scheme. Define the representation before discussing error or speed.

Separate rounding and clipping

With zero point zero, integer limits minus seven through seven, and scale one seventh, the grid spans minus one to one. Nearest rounding without clipping gives error no greater than roughly half a step. An input two instead clips to reconstructed one, producing error one. The rounding bound cannot be transferred to saturated values. This symmetric example uses fifteen levels, not necessarily every bit pattern available in a four-bit container. Whether zero is exactly representable and how unused codes are handled depend on the chosen representation and implementation.

Inspect outlier-driven scaling

For values −0.1, zero,0.1, and ten, scaling to maximum magnitude ten gives step ten sevenths. The three small values may collapse to zero. Preserving an outlier can therefore sacrifice resolution for common small values. Narrower clipping improves small-value resolution while increasing the outlier’s error. Which tradeoff is useful depends on frequencies and downstream roles. Calibration data must represent the intended use, and settings chosen without deployment outliers may fail later. Report both mean and maximum error so an average does not conceal a severely distorted but important component.

Compare scale granularity

A quantizer can share one scale across a tensor or use scales per output channel or smaller group. Finer granularity can accommodate differing distributions but adds metadata and computation. Weight-only quantization differs from quantizing activations or the KV cache. Saying a model is four-bit does not establish that every tensor is stored and computed in four bits. Record targets, granularity, reconstruction timing, and accumulation dtype. These details determine both error behavior and implementation cost, and prevent a storage label from being mistaken for a complete description of numerical execution.

Evaluate more than reconstruction

Small reconstruction mean squared error can still alter a classification boundary or the ranking of critical generation candidates. Conversely, some large numeric errors may have little effect on a particular task. Compare the original and modified model on fixed evaluation data using relevant loss, correctness, and output constraints. Separate scale-selection data from final evaluation. Mixed-precision exceptions can preserve sensitive layers, but document them explicitly. Reconstruction error characterizes a representation change; task quality requires another measurement, so neither a favorable tensor statistic nor a bit-width label alone proves that the intended behavior was preserved.

Do not infer speed from bit width alone

Putting integers in an ordinary Python list does not create a packed four-bit GPU representation. Runtime depends on compatible kernels, reconstruction, transfer, batch shape, and hardware. Run the companion small quantization experiment and explain how scales and outliers change reconstruction error. Completion requires recognizing that this arithmetic model does not measure deployed-model quality or GPU throughput. Continue to the quantization-versus-pruning comparison to identify what each method removes, then measure actual memory and latency separately on an implementation that genuinely uses the intended representation.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 18

Choose a learning path