UAT: Why MLPs Can Approximate Continuous Functions

“Fit anything” is not an unconditional statement. A precise version names a function space, an error norm, a domain, and regularity conditions. A rich enough family is dense in that space, meaning the distance between a target f and a model g can be made smaller than any epsilon.

For every f in a function space F and every epsilon > 0,
there exists g in model class G such that

    distance(f, g) < epsilon.

Fourier series and UAT are both instances of this template. The difference is that Fourier bases are usually fixed, while MLP hidden units learn their location, direction, scale, and output weight.

Let K be a compact subset of R^d, and let C(K) be the space of continuous functions on K with the sup norm. For suitable activations, such as continuous non-polynomial activations, or bounded continuous nonconstant sigmoidal activations in Cybenko’s original theorem, single-hidden-layer networks are dense in C(K).

Given f in C(K) and epsilon > 0,
there exist N, alpha_j, w_j, b_j such that

sup_{x in K} | f(x) - sum_{j=1}^N alpha_j sigma(w_j^T x + b_j) | < epsilon.

The key word is existence. UAT says a wide enough MLP is expressive enough. It does not say training will find the parameters or that finite-sample generalization will be good.

The proof outline is functional-analytic. If MLPs were not dense, a nonzero measure would annihilate every hidden unit. But sigmoidal units approximate half-space indicators, forcing the measure of every half-space to be zero. That implies the measure itself is zero, a contradiction.

L(h) = int_K h(x) d mu(x)

Therefore for every w and b:
int_K sigma(w^T x + b) d mu(x) = 0.

sigma(lambda(w^T x + b)) -> 1{w^T x + b > 0}

So, for almost every boundary shift b:
mu({x in K: w^T x + b > 0}) = 0.

hat_mu(xi) = int_K exp(i xi^T x) d mu(x)
          = int_R exp(i t) d nu_xi(t)
          = 0 for every xi.

Thus mu = 0, contradicting that L was nonzero.

For ReLU, the most concrete derivation starts in one dimension. The unit (x-t)_+ is a hinge that changes slope at t. Adding many hinges places many breakpoints, representing any continuous piecewise-linear function.

p(x) = beta_0 + beta_1 x + sum_{i=1}^{m-1} c_i (x - t_i)_+

If slopes on intervals are s_1, s_2, ..., s_m,
then c_i = s_{i+1} - s_i.

A continuous function on a closed interval is uniformly continuous. With a fine enough grid, the polygonal interpolation p satisfies sup_x |f(x)-p(x)| < epsilon. Since p is a finite linear combination of ReLU hinges, the constructive intuition is direct: more hidden units mean more breakpoints.

Fourier series use a fixed orthogonal basis, sin(kx) and cos(kx), on a periodic interval. For square-integrable periodic functions, the partial sum is a projection onto a finite-frequency subspace.

f(x) ~ a_0/2 + sum_{k=1}^N [a_k cos(kx) + b_k sin(kx)]

a_k = (1/pi) int_{-pi}^{pi} f(x) cos(kx) dx
b_k = (1/pi) int_{-pi}^{pi} f(x) sin(kx) dx

Orthogonality gives a clean error decomposition. With complex coefficients c_k=(1/2pi) int f(x)e^{-ikx}dx, Parseval decomposes energy by frequency; truncation error is the energy left in the discarded high frequencies.

||f - S_N f||_2^2 = 2pi * sum_{|k| > N} |c_k|^2

The precise one-sentence summary is: Fourier expansions and UAT both show that complex functions can be approximated by sums of simple functions; Fourier uses fixed orthogonal frequencies, while MLPs use learnable nonlinear units. The approximation idea is shared, but the assumptions, norms, computation, and failure modes differ.