4
$\begingroup$

What are the characteristics which make a function difficult for the Neural Network to approximate?

Intuitively, one might think uneven functions might be difficult to approximate, but uneven functions just contain some high frequency terms (which in case of sigmoid is easy to approximate $\frac{1} {(1 + e ^ {-(w*x + b)})}$, by increasing the value of $w$). So uneven data might not be diffcult to approximate.

So my question is what makes a function truly difficult for approximation?

NOTE: By approximation I do not mean things which can be changed by changing the training method (changing training set size, methods, optimisers). By approximation I mean things which require hyperparameters (size, structure, etc) of a NN to be changed to approximate to a certain level significantly easily.

Improve this question
$\endgroup$
4
  • $\begingroup$ What do you mean by "difficult"? Do you mean the time required to approximate? How do you count time? I think that the performance of a NN at approximating a function also depends on the available training data, the training algorithm, etc. So, I think you should clarify what you mean by "difficult". After having quickly looked at the literature, people have been trying to study the approximation properties of NNs also when e.g. weights are fixed but letting the number of neurons vary. $\endgroup$
    – nbro
    Commented Mar 9, 2019 at 9:58
  • $\begingroup$ @nbro I was also thinking the same thing.. Difficult to train vs difficult to approximate...I'll edit the question after some time. $\endgroup$
    – user9947
    Commented Mar 9, 2019 at 9:59
  • 1
    $\begingroup$ The paper Deep Neural Network Approximation Theory might be useful, even though it is highly mathematical, and so it might be not trivial to follow. $\endgroup$
    – nbro
    Commented Mar 9, 2019 at 10:14
  • 1
    $\begingroup$ Possible duplicate of Which functions can't neural networks learn efficiently? $\endgroup$
    – nbro
    Commented May 10, 2019 at 14:45

0

Reset to default

You must log in to answer this question.