Integer-Arithmetic-Only Neural Network Inference — Linear Quantization

Storage alone isn’t enough

In the previous post we looked at K-Means-based quantization. It shrank the model a lot by grouping weights into a few representatives, but it had a decisive limitation: what shrinks is only the storage size, and at inference it decodes through the codebook and still does floating-point arithmetic.

This time we break through that wall. Beyond storing weights as integers, we run the multiplications and additions entirely in integer arithmetic — this is Linear Quantization. Proposed in the 2018 paper by Jacob et al.¹, it is exactly what TensorFlow Lite’s INT8 quantization uses.

---

The core: an affine map connecting integers and reals

Linear Quantization is an affine mapping that sends integers to reals. It’s just one line.

$$r = S \cdot (q - Z)$$

• $r$ — the original real value (floating-point)
• $q$ — the quantized integer
• $Z$ — the zero point: the integer that corresponds exactly to the real value $0$. (integer)
• $S$ — the scale: how much one integer step is worth in reals. (floating-point)

So from an integer $q$, subtract the zero point $Z$ and multiply by the scale $S$, and the original real is recovered. Conversely, to turn a real into an integer, round: $q = \text{round}(r / S + Z)$.

Why $Z$ exists separately matters. The real value $0$ (e.g. after ReLU, or padding) is extremely common in networks and must be represented exactly. The zero point is the device that guarantees real 0 lands on some integer with zero error.

Check it in the widget below. A real weight matrix is quantized to integers $q$, then reconstructed via $S(q-Z)$. Changing bits changes the integer range; pressing asymmetric/symmetric changes how $Z$ is handled; and the scale $S$, zero point $Z$, and quantization error below update in real time.

---

How are S and Z determined?

The range of an $N$-bit integer is set by two’s complement.

Bit width	$q_{\min}$	$q_{\max}$
2	−2	1
3	−4	3
4	−8	7
$N$	$-2^{N-1}$	$2^{N-1}-1$

Now let the ends of the real range $[r_{\min}, r_{\max}]$ correspond to the ends of the integer range.

$$r_{\max} = S(q_{\max} - Z), \qquad r_{\min} = S(q_{\min} - Z)$$

Subtracting the two makes $Z$ vanish and gives the scale.

$$S = \frac{r_{\max} - r_{\min}}{q_{\max} - q_{\min}}$$

For the widget default (2-bit, real range $[-1.08, 2.12]$) this gives $S = \frac{2.12 - (-1.08)}{1 - (-2)} = \frac{3.20}{3} \approx 1.07$.

The zero point comes from the same equation. Solve $r_{\min} = S(q_{\min} - Z)$ for $Z$, and round since it must be an integer.

$$Z = \text{round}\left(q_{\min} - \frac{r_{\min}}{S}\right)$$

For the same example, $Z = \text{round}\left(-2 - \frac{-1.08}{1.07}\right) = \text{round}(-0.99) = -1$. Check this against the $S$ and $Z$ shown in the widget’s stats.

---

The real core: integer matrix multiplication

So far this is “storing as integers”. Now let’s see the part where compute is also integer. Consider the fundamental operation, the matrix product $Y = WX$. Substituting the affine map for each value:

$$S_Y(q_Y - Z_Y) = S_W(q_W - Z_W)\cdot S_X(q_X - Z_X)$$

Solving for $q_Y$:

$$q_Y = \frac{S_W S_X}{S_Y}\big(q_W q_X - Z_W q_X - Z_X q_W + Z_W Z_X\big) + Z_Y$$

Look inside the parentheses. $q_W q_X$ is an integer multiplication, and the remaining terms $Z_W q_X$, $Z_X q_W$, $Z_W Z_X$ are all products and sums of integers. Moreover, terms independent of the input, like $Z_W Z_X$, can be precomputed. So the whole parenthesis finishes in integer arithmetic.

What’s left is only the leading $\frac{S_W S_X}{S_Y}$, which is floating-point. Here’s the trick of the method. Empirically this scale ratio is always in $(0, 1)$, so it can be written as:

$$\frac{S_W S_X}{S_Y} = 2^{-n} M_0, \qquad M_0 \in [0.5, 1)$$

$M_0$ is a fixed-point multiplication, and $2^{-n}$ is just a bit shift. In the end no floating-point unit is needed at all. Every operation finishes as integer multiplication, addition, and shift.

---

Simpler still: symmetric quantization and bias folding

In practice two simplifications are used.

Symmetric quantization — weight distributions are usually symmetric about 0. So we fix the weights’ zero point at $Z_W = 0$. Then the $Z_W q_X$ and $Z_W Z_X$ terms above vanish, making it much cleaner.

$$q_Y = \frac{S_W S_X}{S_Y}\big(q_W q_X - Z_X q_W\big) + Z_Y$$

Bias folding — for $Y = WX + b$ with bias, setting $Z_b = 0$ and $S_b = S_W S_X$ absorbs the bias naturally. Precomputing the input-independent part ($q_b - Z_X q_W$) into a single $q_{bias}$:

$$q_Y = \frac{S_W S_X}{S_Y}\big(q_W q_X + q_{bias}\big) + Z_Y$$

A convolution layer has the same structure — just $\text{Conv}(q_W, q_X)$ in place of $q_W q_X$. In summary, the integer inference pipeline becomes:

1. integer MAC (multiply-accumulate) on integer inputs/weights → int32 accumulation
2. integer addition of the precomputed $q_{bias}$
3. rescale $\frac{S_W S_X}{S_Y}$ via fixed-point multiply + shift → to an N-bit integer
4. integer addition of the zero point $Z_Y$ → integer output

There is no floating-point anywhere.

---

Results: how much is lost, how much faster

Even quantized to INT8 this way, accuracy is largely preserved.

Model	Float accuracy	INT8 accuracy
ResNet-50	76.4%	74.9%
Inception-V3	78.4%	75.4%

With around 1–3%p loss, the model is 1/4 the size and the compute is integer. On mobile (Snapdragon), integer-only inference shows far lower latency than float at the same accuracy — because integer units are cheaper and faster. (The point from the first post, “low-bit integer ops are tens of times cheaper than float,” becomes real here.)

---

Summary

• Affine map $r = S(q - Z)$ connects the integer $q$ and the real $r$. $Z$ (zero point) makes real 0 land on an integer with zero error; $S$ (scale) is the size of one step.
• $S = \frac{r_{\max}-r_{\min}}{q_{\max}-q_{\min}}$, $Z = \text{round}(q_{\min} - r_{\min}/S)$.
• Integer matmul: $q_W q_X$ plus precomputed terms make the parenthesis integer arithmetic; the leading scale ratio is a fixed-point multiply + shift. → inference with no floating-point.
• Symmetric quantization ($Z_W=0$) and bias folding make the formula even simpler.

The biggest difference is here. K-Means quantization was integer in storage only (compute was float). Linear Quantization is integer in storage and integer in compute.

Method	Storage	Compute
Original	FP weights	FP arithmetic
K-Means quantization	integer indices + FP codebook	FP arithmetic
Linear quantization	integer weights	integer arithmetic

With this we’ve seen both major branches of neural network quantization — K-Means, which reduces storage, and Linear, which makes even the compute integer. Next comes the story of pushing it to the extreme: Binary and Ternary quantization, shrinking weights to a single $+1/-1$ bit.