Recently, Kolmogorov-Arnold Networks (KANs) generated a lot of hype, with "AI experts" throwing around terms like "ML 2.0" and "a new era of ML".
KANs are supposedly inspired by the Kolmogorov-Arnold representation theorem, which states that any continuous multivariate function can be represented as a superposition of continuous functions of a single variable and addition. Formally, it states that any continuous function $f: [0, 1]^n \to \mathbb{R} $ can be written as:
$$ f(x_1, x_2, \ldots, x_n) = \sum_{i=0}^{2n} \phi_i \left( \sum_{j=1}^n \psi_{ij}(x_j) \right) $$ where $\phi_i$ and $\psi_{ij}$ are continuous functions.
From my current understanding , this implies that any multivariate continuous function can be represented by
- Transforming each variable independently with a continuous (univariate) function ($\psi_{ij}(x_j)$)
- Summing the transformed values
- Transforming this sum with another continuous univariate function ($\phi_i$)
- (and since for some reason that I don't currently understand the process above happens i times) finally summing all these i terms to obtain a single result
It seems to me like MLPs would actually be a simple example of this:
- In the linear layer, in one neuron, each input is multiplied by a weight, i.e. $\psi_{ij}(x_j) = w_j x_j$
- These terms are summed: $w_1 x_1 + w_2 x_2 + … + w_d x_d$
- The sum is transformed by an activation function such as a ReLU: $\text{ReLU}(w_1 x_1 + w_2 x_2 + … + w_d x_d)$, i.e. $\phi_i = \text{ReLU}$
- (In this case it seems to me that step 4 is not necessary, but I might be wrong.)
Is this correct or am I missing something? And if so: Why do people seem to say that MLPs are more closely connected to the Universal Approximation Theorem (see e.g. this post), when Kolmogorov-Arnold seems to be so related?
