What are the characteristics which make a function difficult for the Neural Network to approximate?

Intuitively, one might think uneven functions might be difficult to approximate, but uneven functions just contain some high frequency terms (which in case of sigmoid is easy to approximate $\frac{1} {(1 + e ^ {-(w*x + b)})}$, by increasing the value of $w$). So uneven data might not be diffcult to approximate.

So my question is what makes a function truly difficult for approximation?

NOTE: By approximation I do not mean things which can be changed by changing the training method (changing training set size, methods, optimisers). By approximation I mean things which require hyperparameters (size, structure, etc) of a NN to be changed to approximate to a certain level significantly easily.

  • $\begingroup$ What do you mean by "difficult"? Do you mean the time required to approximate? How do you count time? I think that the performance of a NN at approximating a function also depends on the available training data, the training algorithm, etc. So, I think you should clarify what you mean by "difficult". After having quickly looked at the literature, people have been trying to study the approximation properties of NNs also when e.g. weights are fixed but letting the number of neurons vary. $\endgroup$
    – nbro
    Mar 9 '19 at 9:58
  • $\begingroup$ @nbro I was also thinking the same thing.. Difficult to train vs difficult to approximate...I'll edit the question after some time. $\endgroup$
    – user9947
    Mar 9 '19 at 9:59
  • 1
    $\begingroup$ The paper Deep Neural Network Approximation Theory might be useful, even though it is highly mathematical, and so it might be not trivial to follow. $\endgroup$
    – nbro
    Mar 9 '19 at 10:14
  • 1
    $\begingroup$ Possible duplicate of Which functions can't neural networks learn efficiently? $\endgroup$
    – nbro
    May 10 '19 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy