Why are there suddenly so many data types?
A few years ago, float32 was all deep learning needed. But open any recent paper or model card and you’re buried in data types: FP16, BF16, FP8 (E4M3/E5M2), INT8, INT4, FP4, NF4… The names alone don’t tell you what’s what, or why there are so many.
The reason is simple: low-bit operations are just cheap. Here is the rough per-operation energy cost in a 45nm process.
| Operation | Energy (pJ) |
|---|---|
| 8-bit int ADD | 0.03 |
| 32-bit int ADD | 0.1 |
| 32-bit float ADD | 0.9 |
| 8-bit int MULT | 0.2 |
| 32-bit float MULT | 3.7 |
An 8-bit integer multiply is about 18× cheaper than a 32-bit float multiply, and addition is 30× cheaper. On top of that, reading a single value from memory costs hundreds of times more than one multiply or add. So for the same model, the fewer bits you store each number in, the more power, memory, and bandwidth you save. All those data types above are just different answers to one question: “how many bits, and where do we spend them?”
The catch is that fewer bits means fewer distinct numbers you can represent, and you lose precision or range accordingly. So each format has its own character depending on how it splits its bits among sign, exponent, and fraction. This post takes those splitting rules apart — how bits are actually interpreted as numbers — one by one.
For each data type there’s a widget where you can click the bits directly. Flip the 0s and 1s and the formula and result update in real time. Poking at the bits yourself sinks in a lot faster than reading an explanation.
1. Integer
Unsigned Integer
The simplest. Each of the $n$ bits carries a place value $2^k$, and you sum the place values of the bits that are on ($=1$).
$$\text{value} = \sum_{i=0}^{n-1} b_i \cdot 2^i$$The range is $[0,\ 2^n - 1]$. For 8 bits, that’s 0 to 255.
Click the bits in the widget below. The default 00110001 is $2^5 + 2^4 + 2^0 = 49$.
Signed Integer
To represent negatives you need a sign. There are two approaches.
Sign-Magnitude — use the leading bit as a sign ($0$=positive, $1$=negative) and the rest for magnitude. Intuitive, but it has a fatal flaw: 00000000 and 10000000 are both zero (+0 and −0). Two zeros make the hardware awkward. Range: $[-2^{n-1}+1,\ 2^{n-1}-1]$.
Two’s Complement — what modern computers actually use. The idea is simple: make the leading bit’s place value negative. That is, the MSB’s weight is $-2^{n-1}$ instead of $+2^{n-1}$.
$$\text{value} = -b_{n-1}\cdot 2^{n-1} + \sum_{i=0}^{n-2} b_i \cdot 2^i$$Now zero is only 00000000, and the range $[-2^{n-1},\ 2^{n-1}-1]$ reaches one further on the negative side. Below, 11001111 is $-2^7 + 2^6 + 2^3 + 2^2 + 2^1 + 2^0 = -49$. Try turning the MSB off and see what happens.
2. Fixed-Point
Integers alone can’t hold fractions. The easiest extension is to fix the position of the point. Read the bit string as an ordinary two’s-complement integer, then scale it by a fixed $2^{-f}$.
$$\text{value} = (\text{integer value}) \times 2^{-f}$$Below splits 8 bits into a 4-bit integer part and a 4-bit fractional part ($f=4$). The bit weights run $2^3, 2^2, \dots, 2^0, 2^{-1}, \dots, 2^{-4}$. 00110001 reads as the integer 49, so $49 \times 2^{-4} = 3.0625$.
Fixed-point is simple but clearly limited. Because the point is fixed, the range of magnitudes it can represent is narrow — it struggles to handle very large and very small numbers at the same time. Solving that is the job of our next subject: floating-point.
3. Floating-Point — IEEE 754
In floating-point, as the name says, the point floats. Think of it as the binary version of scientific notation ($1.5 \times 10^3$). The bits split into three parts.
- Sign — 1 bit
- Exponent — sets the scale of the magnitude → determines range
- Fraction — sets the fine value → determines precision
The formula for a normal number:
$$\text{value} = (-1)^{\text{sign}} \times (1 + \text{Fraction}) \times 2^{\text{Exponent} - \text{bias}}$$Here bias is an offset that lets the exponent go negative, equal to $2^{e-1}-1$ ($e$ = number of exponent bits). And note the $(1 + \cdots)$ in front of the fraction: a normal number always carries an implicit leading 1 (the “implicit leading 1”).
A quick note on terminology — Fraction / Mantissa / Significand
These terms get mixed up across sources. The whole $1.\text{Fraction}$ in the formula — the significant-digits part — is formally called the Significand.
- Fraction — refers only to the fractional part actually stored in the bits. In the widget above, that’s the 23 bits colored yellow. It’s the value below the point, $0.\text{b}_1\text{b}_2\cdots$.
- Significand — the whole $1.\text{Fraction}$ including the implicit 1. This is the actual significand that gets multiplied.
- Mantissa — historically the fractional part of a logarithm table; in floating-point it’s used loosely as a synonym for Fraction. The IEEE 754 standard itself uses “significand” as the official term and discourages “mantissa,” but in practice “mantissa” is still everywhere.
In short, Significand = 1 + Fraction, and when people casually say “mantissa” they usually mean the stored Fraction. In this post we call the stored field the Fraction.
Below is 32-bit single precision (FP32). Sign 1 + exponent 8 + fraction 23 = 32 bits, bias 127. The default represents $0.265625 = (1 + 0.0625) \times 2^{125-127}$. Click the exponent bits one at a time and watch the value double each step.
Special values: zero, infinity, NaN, and subnormals
There’s something odd about the formula: because of $(1 + \text{Fraction})$, a normal number can’t represent zero. So IEEE 754 uses the exponent field as a special signal.
| Exponent | Fraction = 0 | Fraction ≠ 0 | Interpretation |
|---|---|---|---|
00…0 (=0) | $\pm 0$ | subnormal | $(-1)^s \times \text{Fraction} \times 2^{1-\text{bias}}$ |
00…1 ~ 11…0 | normal | normal | $(-1)^s \times (1+\text{Fraction}) \times 2^{\text{Exp}-\text{bias}}$ |
11…1 (=max) | $\pm\infty$ | NaN | — |
The key case is when the exponent is all zeros. Then it drops the implicit 1 ($1+\text{Fraction} \to \text{Fraction}$) and fixes the exponent at $2^{1-\text{bias}}$. This produces subnormal numbers, which densely fill in the tiny values near zero. Conversely, when the exponent is all ones you get infinity (fraction 0) and NaN (fraction ≠ 0).
Make them yourself in the FP32 widget above:
- Set the exponent to all ones (
0 11111111 0…0) →+∞ - Then turn on any fraction bit →
NaN - All zeros →
0 - Leave the exponent at zero and turn on a few fraction bits → a tiny subnormal value
The wider the exponent, the wider the range; the wider the fraction, the higher the precision. Exponent → Range, Fraction → Precision. That one line is the core trade-off behind every low-precision format that follows.
4. Half the size: FP16 and BF16
FP32 is accurate but eats 32 bits. Deep learning often doesn’t need that much precision, so we use 16-bit formats. Same 16 bits, but where you spend them differs.
FP16 (IEEE 754 Half Precision)
Exponent 5 + fraction 10, bias 15. A split that invests more in precision (the fraction). Below is $1\,10001\,1100000000$, i.e. $-(1+0.75)\times 2^{17-15} = -7.0$.
BF16 (Google Brain Float)
Exponent 8 + fraction 7, bias 127. Same total bit count as FP16, but it keeps the exponent at 8 bits, exactly like FP32. So its range is identical to FP32, at the cost of precision. It’s popular because values whose scale swings wildly — like gradients during training — won’t overflow.
Below represents $2.5 = (1 + 0.25)\times 2^{1}$ in BF16 ($0\,10000000\,0100000$).
Feel the difference in exponent width between the two widgets. BF16 has 8 exponent cells, so it reaches very large and very small numbers, but with only 7 fraction cells its values are sparse. FP16 is the opposite.
5. Going lower: FP8 and FP4
FP8 (E4M3 / E5M2)
8-bit floating-point is supported by the latest hardware (e.g. Nvidia Hopper/Blackwell). Two splits are used as de facto standards.
- E4M3 — exponent 4 + fraction 3. Precision-first. Mainly for weights and activations in the forward pass. It has no INF and uses only
S.1111.111for NaN. The largest representable normal value is $448$. - E5M2 — exponent 5 + fraction 2. Range-first. Used for large-scale values like gradients in the backward pass. It has INF and NaN like IEEE.
E4M3 first. Bias is 7. The default 0 0111 000 is $(1+0)\times 2^{7-7} = 1.0$.
E5M2. Bias 15. Five exponent cells hold a much wider range, but with only 2 fraction cells the values are spaced far apart.
INT4 and FP4
The extreme. Four bits can represent just 16 values. How you place those 16 differs by format.
INT4 — two’s-complement integer. Laid out at uniform spacing from $-8$ to $7$.
FP4 distributes its values differently depending on the exponent/fraction split. The more exponent bits, the more the values cluster densely near zero and spread out sparsely far away — a non-uniform spacing.
- E1M2 — exponent 1 + fraction 2. Closest to an integer (bias 0).
0111= $(1+0.75)\times 2^{1-0} = 3.5$.
- E2M1 — exponent 2 + fraction 1 (bias 1).
0111= $(1+0.5)\times 2^{3-1} = 6$.
- E3M0 — exponent 3 + fraction 0 (bias 3). With no fraction, it essentially represents only powers of two.
0111= $(1+0)\times 2^{7-3} = 16$. Hit the random button and watch the values spread out exponentially, like $\dots, 4, 8, 16$.
This leads to an interesting observation. A neural network’s weights are usually clustered near zero with long tails. If so, a floating-point format that packs values densely near zero can fit the actual values better than a uniform INT, even at the same 4 bits. Which distribution suits which format — that’s why the choice of data type feeds directly into accuracy.
Wrapping up
That’s the story of what the data types you meet in deep learning actually mean. To recap:
- Integer / fixed-point — uniform spacing. Simple, but narrow range.
- Floating-point — splits its bits between exponent (range) and fraction (precision). Dense near zero, sparse far out.
- Cutting bits = cutting the number of representable values. From FP32’s ~4.3 billion down to FP4’s 16.
Now when you see names like FP16, BF16, FP8 E4M3, INT4, you’ll picture the bit layout in your head. In deep learning, “which data type should I use?” is no longer a trivial implementation detail — it’s a design choice that decides how small, how fast, and yet how accurately you can run your model.