Signal Channel and Reservoir

A small MLP trained on noisy 1D regression. The empirical NTK spectrum splits into a signal channel (large eigenvalues, fast learning) and a reservoir (small eigenvalues, where memorized noise sits).

Width80

Train points N30

Noise std0.18

Learning rate0.050

Steps / frame20

Network function f(x)

true signal noisy training data network test points

Loss curves

train loss test loss

Step0

Train—

Test—

Empirical NTK eigenspectrum on training set

signal channel (large λ) reservoir (small λ, near-null)

Vertical bars show eigenvalues of K_SS = J_S J_S^⊤ at the current parameters, sorted descending. The dashed line marks the boundary λ ≈ max(λ)/100. The histogram below shows how the residual r = f(x) − y projects onto each eigenvector. Memorized noise concentrates in the reservoir, where the test transfer cancels it.