5
$\begingroup$

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable to me, because something like Fourier series are also able to approximate almost any function between compact domains of Euclidean spaces.

So my question is, what makes neural networks different from something like Fourier analysis where we can approximate any sufficiently nice function we like as well?

Am I not understanding the universal approximation theorem, or are there justifications for the power of neural networks that go deeper than talk about approximation?

$\endgroup$
1

2 Answers 2

4
$\begingroup$

People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data.

There is an opinion that this is completely missing the point. It doesn't really matter how well you can approximate your data - a lookup table that completely remembers your data can perform with 100% accuracy. What matters is the generalization capabilities of your functional family: how well does it interpolates and extrapolates beyond its training observations.

Speaking about the Fourier transform - I think one can even say that the Gibbs phenomenon can be considered as very early example of people noticing such "bad generalization" behavior.

Why deep neural networks are so (comparatively) good at generalization? There's still a discussion about it. There's Manifold hypothesis, there's kernel machines interpretation, there are studies of training landscapes of deep neural nets. There are even takes that employ criticality analysis from physics. To me it looks like all these approaches are parts of a bigger picture that we are yet to see.

$\endgroup$
3
$\begingroup$

It's worth noting that the Fourier series analogy was used in early explorations of universal approximation theorems https://ieeexplore.ieee.org/document/23903.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .