Fix a simple grid
Quantize values −1,−0.3, zero,0.2, and one onto a symmetric integer grid. With limits minus m through m, scale is maximum absolute value divided by m; round x over scale, clip, then reconstruct by multiplying by scale. Zero point is zero. Limits seven and one hundred twenty-seven provide fifteen and two hundred fifty-five levels, respectively, rather than specifying every bit pattern of a storage format. Python round uses ties-to-even. Match rounding conventions when comparing implementations, and write the grid definition before interpreting a reported bit width.
Run without special hardware
Run the code with standard Python 3 after saving it as quantize.py. Inspect scale, integer values, reconstructed values, and maximum absolute error. Ordinary Python integer lists are not packed four-bit or eight-bit arrays. This is an experiment on a numeric mapping, not GPU memory savings or speed. It requires no model download, network, or additional package and changes no external dataset. Cleanup is limited to the saved script. Keeping these boundaries explicit lets a CPU exercise teach quantization error without implying implementation or hardware properties that it never measures.
Check the unsaturated rounding bound
The first two settings choose range from the largest absolute input, so no value lies outside it. Assertions check the nearest-rounding bound of half a step with a small tolerance. A larger m makes the grid finer, but each element’s error need not decrease in an identical ratio because some values can coincide with grid points. Add mean squared error if useful alongside maximum error. One exactly reconstructed element does not imply zero error for the entire tensor. The explicit restored list makes it possible to inspect which values, not just which aggregate, changed.
def quantize(values, limit):
scale = max(abs(x) for x in values)/limit or 1.0
integers = [max(-limit, min(limit, round(x/scale))) for x in values]
restored = [q*scale for q in integers]
errors = [abs(a-b) for a,b in zip(values, restored)]
return scale, integers, restored, errors
values = [-1.0, -0.3, 0.0, 0.2, 1.0]
for limit in [7, 127]:
s, q, restored, errors = quantize(values, limit)
print(limit, s, q, restored, 'max error', max(errors))
assert max(errors) <= s/2 + 1e-12
s, q, restored, errors = quantize([-0.1, 0.0, 0.1, 10.0], 7)
assert q[:3] == [0,0,0]
print('outlier scale', s, 'integers', q, 'errors', errors)
scale = 1/7
clipped = min(7, max(-7, round(2.0/scale))) * scale
assert clipped == 1.0
print('clipped input2 restored', clipped)
Introduce an outlier
For −0.1, zero,0.1, and ten, the outlier sets scale to ten sevenths and the first three values all become integer zero. Even their sign distinction is lost. Preserving the outlier range competes with resolving the small values. Narrower clipping or group-specific scales are possible alternatives, but introduce clipping error or additional metadata. This experiment alone cannot determine whether the lost distinctions matter to a task. First describe the numerical change, then evaluate downstream predictions separately instead of treating a tensor-level statistic as a direct measure of model quality.
Test deliberate clipping
Finally hold scale at one seventh and quantize input two with upper integer limit seven. Reconstruction is one and error is one, much larger than half a step. The unsaturated rounding bound does not apply outside the represented range. This can occur when scales fitted on one sample are applied to another. Separate calibration and evaluation data and report the saturation fraction. A small average error can conceal rare severe clipping, so retain range violations rather than silently discarding them or interpreting every discrepancy as ordinary rounding noise.
Report numeric and implementation claims separately
Report both limits’ scales and maximum errors, the outlier example’s integers, and the deliberately clipped reconstruction. Completion means distinguishing resolution from represented range and rounding from saturation. A real model experiment additionally records whether weights, activations, or cache are quantized, scale granularity, accumulation type, and kernels. Do not report Python-list size or execution time as efficiency of a packed GPU format. Continue to the pruning comparison to separate this representational approximation from removing weights or structures, which introduces another kind of implementation and quality tradeoff.