1
$\begingroup$

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following:

Different choices of activation functions can be used to simulate different types of models used in machine learning, like least-squares regression with numeric targets, the support vector machine, or a logistic regression classifier. Most of the basic machine learning models can be easily represented as simple neural network architectures.

I remember reading something about it being mathematically proven that neural networks can approximate any function, and therefore any machine learning method, or something along these lines. Am I remembering this correctly? Would someone please clarify my thoughts?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ See en.wikipedia.org/wiki/Universal_approximation_theorem $\endgroup$
    – pasaba por aqui
    Nov 7, 2020 at 19:16
  • $\begingroup$ @pasabaporaqui Thanks for that. Chapter 1.5 The Secrets to the Power of Function Composition seems to say the following: "This basic idea is the essence of the universal approximation theorem of neural networks. In fact, the proof of the ability of squashing functions to approximate any function is conceptually similar to that of kernels at least at an intuitive level. However, the number of base functions required to reach a high level of approximation can be extremely large in both cases, potentially increasing the data-centric requirements to an unmanageable level. [...] $\endgroup$
    – The Pointer
    Nov 7, 2020 at 19:56
  • $\begingroup$ [...] For this reason, shallow networks face the persistent problem of overfitting. The universal approximation theorem asserts the ability to well-approximate the function implicit in the training data, but makes no guarantee about whether the function can be generalized to unseen test data." $\endgroup$
    – The Pointer
    Nov 7, 2020 at 19:56

1 Answer 1

Reset to default
1
$\begingroup$

I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point.

I don't think there is a relation between this argument and universal approximation theorems, which state that certain classes of neural networks can approximate any function in certain domains, rather than any learning algorithm.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Perhaps I am indeed mixing up two different things. Nonetheless, if a neural network is able to approximate any function, then, given sufficient data (however much "sufficient" may be), it should be able to accomplish anything any other learning algorithm, including the more "conventional" machine learning algorithms, can, right? $\endgroup$
    – The Pointer
    Nov 7, 2020 at 20:01
  • $\begingroup$ "should be able to accomplish anything any other learning algorithm [...] can": there is a subtlety though. Universal approximation only state existence of neural networks that approximate functions, but say nothing about algorithms for finding them. $\endgroup$
    – Dani
    Nov 7, 2020 at 22:14
  • $\begingroup$ But doesn't this comment agree with what I said? Since neural networks are able to approximate any function, then it should (assuming sufficient data) be able to approximate the same functions that any other learning algorithms do? Or am I still misunderstanding? $\endgroup$
    – The Pointer
    Nov 7, 2020 at 22:20
  • 1
    $\begingroup$ Let me put this another way: the functions defined by neural networks with specific weights and architectures set are different from the algorithms used to find them. Universal approximation theorems say: given a target function, a neural network that does not differ from it to a given degree of tolerance exists. However, a classical algorithm might find an acceptable function, whereas you might not be able to find such neural network. About the "sufficient data" question: UA theorems are valid for certain classes of NNs with infinite VC dimension, so infinite sample complexity. $\endgroup$
    – Dani
    Nov 7, 2020 at 22:27
  • 1
    $\begingroup$ An architecture and specific weights for that architecture. Even if your architecture was correct, you still need to find the specific weights. $\endgroup$
    – Dani
    Nov 7, 2020 at 22:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .