People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable to me, because something like Fourier series are also able to approximate almost any function between compact domains of Euclidean spaces.
So my question is, what makes neural networks different from something like Fourier analysis where we can approximate any sufficiently nice function we like as well?
Am I not understanding the universal approximation theorem, or are there justifications for the power of neural networks that go deeper than talk about approximation?