Deep-Learning

A ReLU Network Is One Giant Piecewise-Affine Function

A network with ReLU activations secretly carves input space into polytope pieces and, on each piece, collapses into exactly one affine map. This post builds that view up from a single neuron to interactive demos.

Shrinking Models by Sharing Weights — K-Means-based Quantization

Group a neural network’s weights into a few representative values with K-Means, and you can shrink the model several-fold with almost no accuracy loss. We explore it with an interactive widget where you watch the clusters converge and the storage shrink in real time. (Deep Compression, Han et al. 2016)

Integer-Arithmetic-Only Neural Network Inference — Linear Quantization

Beyond storing weights as integers — running the multiplications and additions entirely in integer arithmetic at inference. We connect reals and integers with the affine map r = S(q − Z), and explore it with a widget where you change the scale and zero point and watch the quantization error. (Jacob et al. 2018, the basis of TFLite integer quantization)

Data Types in the Deep Learning Era

INT8, FP16, BF16, FP8, FP4 — what do the data types you keep seeing in deep learning actually mean, and how do bits turn into numbers? We take them apart one by one, with widgets where clicking a bit updates the formula and value in real time.