A ReLU Network Is One Giant Piecewise-Affine Function

Introduction

We’re taught that a neural network is a “nonlinear function approximator.” True enough. But when the activation is piecewise-linear like ReLU, something far more concrete and surprising holds.

A network built from ReLUs is globally a piecewise-affine function. Input space gets chopped into polytope pieces, and inside each piece all of the deep network’s nonlinearity vanishes — it collapses into exactly one affine map $\mathbf{x} \mapsto W_{\text{eff}}\mathbf{x} + \mathbf{b}_{\text{eff}}$.

This post builds that fact up layer by layer, starting from a single neuron, with interactive demos to check it yourself along the way.

1. One neuron = one boundary

Start with a single ReLU neuron.

This is a two-piece function:

• $\mathbf{w}^\top \mathbf{x} + b < 0$ → output $0$ (off, slope $0$)
• $\mathbf{w}^\top \mathbf{x} + b \ge 0$ → pass the input through (on, slope $1$)

The boundary separating the two is a single hyperplane $\mathbf{w}^\top \mathbf{x} + b = 0$. Think of it as one neuron = one cut drawn across input space.

What happens when we take a linear combination of several? For 1D input, plot $f(x) = \sum_i v_i\,\mathrm{ReLU}(w_i x + b_i)$. Each ReLU introduces one knot (a kink), and between knots the function is a straight line.

The strips below the graph unroll the single hidden layer all these neurons live in. Each row is one neuron, colored over the interval where it is on ($w_i x + b_i > 0$). Every time $x$ crosses a knot, exactly one strip toggles (the colors line up with the knot’s vertical guide), and each straight segment is one activation pattern. So the segments are not different layers — they are different combinations of which neurons are on within the same one layer.

Increase the number of ReLUs and the kinks multiply until it looks like a curve — but zoom in and it’s still just straight segments joined at knots. $n$ ReLUs make $n+1$ pieces. That’s the 1D picture. Now let’s generalize to arbitrary dimension and depth.

2. Activation patterns and diagonal masks

The key trick is to rewrite ReLU as a multiplicative mask. A single layer’s forward pass is

Fix a single input $\mathbf{x}$, and every neuron in this layer is determined to be on (1) or off (0). Pack those on/off bits into a diagonal matrix $D^{(l)}(\mathbf{x})$ — diagonal $1$ for an on neuron, $0$ for an off one:

This is not an approximation — it’s an exact identity, since an on neuron does $z \mapsto z$ (identity) and an off neuron does $z \mapsto 0$. The only subtlety is that the mask $D^{(l)}$ depends on $\mathbf{x}$.

Collect the on/off bits of the whole network (every layer, every neuron) into one vector — the activation pattern. With $N$ hidden neurons total, the pattern is a point in $\{0,1\}^N$.

3. Fix the pattern → collapse to affine

Now consider a region where you can nudge $\mathbf{x}$ around and every neuron keeps the same on/off state. Inside that region every $D^{(l)}$ becomes a constant matrix, so the forward pass through $L$ layers is

and since every $D$ and $W$ is constant, the whole composition collapses into a single affine map:

Here $D^{(l)}W^{(l)}$ is “the weight matrix with the rows of off-neurons zeroed out.” So inside that piece the network is exactly one linear transform plus a shift. All of the deep network’s nonlinearity is gone within the region.

It’s hard to believe from prose alone, so let’s extract the effective affine map $(W_{\text{eff}}, \mathbf{b}_{\text{eff}})$ for the piece containing any given input.

import numpy as np

def relu_net(x, Ws, bs):
    """ReLU MLP forward pass (last layer linear)"""
    a = x
    for i, (W, b) in enumerate(zip(Ws, bs)):
        z = W @ a + b
        a = np.maximum(0, z) if i < len(Ws) - 1 else z
    return a

def effective_affine(x, Ws, bs):
    """Extract the effective affine map (W_eff, b_eff) of x's piece"""
    W_eff = np.eye(len(x))
    b_eff = np.zeros(len(x))
    a = x
    for i, (W, b) in enumerate(zip(Ws, bs)):
        z = W @ a + b
        if i < len(Ws) - 1:
            mask = (z > 0).astype(float)      # this layer's pattern D^(l)
            D = np.diag(mask)
            a = mask * z
        else:
            D = np.eye(len(z))                # final linear layer
            a = z
        W_eff = D @ W @ W_eff                 # W_eff <- D W W_eff
        b_eff = D @ (W @ b_eff + b)           # accumulate bias
    return W_eff, b_eff

# one random network
rng = np.random.default_rng(0)
dims = [4, 16, 16, 3]
Ws = [rng.normal(size=(dims[i+1], dims[i])) for i in range(len(dims)-1)]
bs = [rng.normal(size=dims[i+1]) for i in range(len(dims)-1)]

x = rng.normal(size=4)
W_eff, b_eff = effective_affine(x, Ws, bs)

# check 1: exact match at that point
print(np.allclose(relu_net(x, Ws, bs), W_eff @ x + b_eff))   # True

# check 2: still matches at a nearby point in the same piece
x2 = x + 1e-4 * rng.normal(size=4)
print(np.allclose(relu_net(x2, Ws, bs), W_eff @ x2 + b_eff)) # True (same pattern)

W_eff @ x + b_eff matches the real forward pass exactly. And the same $(W_{\text{eff}}, \mathbf{b}_{\text{eff}})$ keeps working for other points in the same piece. Cross a piece boundary (the pattern flips) and you switch to a different affine map.

4. The pieces are polytopes; the whole is continuous piecewise-affine

Each neuron’s on/off condition is a single inequality (that neuron’s pre-activation $\ge 0$). Fixing an activation pattern means simultaneously satisfying $N$ such linear inequalities, and an intersection of linear inequalities is a convex polytope.

Note: A first-layer neuron’s boundary is a straight hyperplane, but a later neuron’s pre-activation is computed through earlier ReLUs, so its boundary is a bent, piecewise-linear surface. Even so, inside each piece everything is linear, so the final regions are still (convex) polytopes.

The full picture, then:

• Input space $\mathbb{R}^d$ is tiled by polytope pieces.
• On each piece the function is a distinct affine map $W_{\text{eff}}\mathbf{x} + \mathbf{b}_{\text{eff}}$.
• Crossing a boundary doesn’t make the value jump, because ReLU is continuous → continuous.

Together this is a CPWL (Continuous Piecewise-Linear) function. And a theorem: the functions a ReLU MLP can represent = exactly the CPWL functions. Not broader, not narrower — precisely that class.

The demo below actually tiles a 2D input space. For a randomly initialized small ReLU network, pixels with the same activation pattern get the same color. Every color boundary is some neuron’s on/off boundary.

• Hover over it. The piece under the cursor (same activation pattern) lights up. Across that entire bright region, the network is a single affine map.
• Increase the width (neuron count) and more cuts appear, splitting the space into finer pieces.
• Increase the number of hidden layers. The boundaries change from straight lines to bent lines, and the piece count explodes.

5. What this view buys you

A surprising number of deep-learning properties fall out of this one perspective.

The gradient is piecewise-constant

Inside a piece the Jacobian is constant, equal to $W_{\text{eff}}$. All backprop does is hand back “the effective linear map of the piece the input landed in.” ReLU’s $0/1$ derivative is the mask $D^{(l)}$. At piece boundaries the derivative jumps discontinuously — which is why a ReLU network’s loss surface is piecewise-smooth.

Depth’s power = an explosion of pieces

Growing depth rather than width can make the number of linear regions grow exponentially in depth (Montúfar et al., 2014). The jump in piece count when you take the demo from 1 → 2 → 3 hidden layers is a miniature of exactly this. It’s the classic expressivity argument for why deep beats shallow.

Layers as folding

Each layer can be read as a folding operation on input space. ReLU folds the negative region down onto $0$, and the next layer draws fresh cuts on the already-folded space. Deeper means more compounded folds and richer piece structure — which is why boundaries look like “bent lines” from the second layer on.

The spline / tropical view

In Balestriero & Baraniuk’s max-affine spline framework, a ReLU network is a multidimensional spline. And since $\max$ and addition correspond to the “addition” and “multiplication” of the tropical semiring, ReLU networks are exactly described by tropical geometry (network = difference of two tropical polynomials, Zhang et al., 2018), opening a path to count pieces via the Newton polytope of a polynomial.

Closing

For ReLU, the sentence “a neural network is a nonlinear function approximator” sharpens into “a neural network is a giant lookup table that partitions input space into polytopes and lays an affine map over each piece.” The nonlinearity lives only in choosing which piece you’re in (i.e. deciding the activation pattern); once the piece is fixed, everything else is pure linear algebra.

This isn’t just an intellectual curiosity. It lets you quantify expressivity via the number of linear regions, locally interpret and verify a model through each region’s effective affine map (e.g. verifiable robustness, exact Jacobian analysis), and carry mature mathematics like spline theory and tropical geometry straight into deep learning. Next time you look at a ReLU network, don’t picture a smooth surface — picture a folded sheet of paper stitched together from countless flat polytope facets.

References

• Montúfar, Pascanu, Cho, Bengio (2014). On the Number of Linear Regions of Deep Neural Networks. NeurIPS.
• Raghu, Poole, Kleinberg, Ganguli, Sohl-Dickstein (2017). On the Expressive Power of Deep Neural Networks. ICML.
• Balestriero, Baraniuk (2018). A Spline Theory of Deep Networks. ICML.
• Zhang, Naitzat, Lim (2018). Tropical Geometry of Deep Neural Networks. ICML.
• Hanin, Rolnick (2019). Complexity of Linear Regions in Deep Networks. ICML.