1
$\begingroup$

Background: I have seen lots of people asking whether multiplication and pseudo-random sequences can be approximated by a NN without providing whether the inputs and outputs are bounded or not, and people have answered it (lot of upvotes) based on conventional NN knowledge. without taking into consideration the aforementioned fact.

TL;DR How good/Is it possible by a Neural Network to approximate an unbounded function provided it is trained on a subset of the number line and the test inputs are significantly outside the subset?

Can a Neural Network do regression for an unbounded function? To me it is impossible if the output function is sigmoid, since the best approximation (basis of all signal decomposition and reconstruction schemes) of a function the Fourier Series ($\star$) demands Dirichlet's condition to be satisfied, one of which more or less states that the value should be absolutely integrable ($\int_{-\infty}^{\infty}|f(x)|^2dx < \infty$). The sigmoid can be more or less thought in terms of a sinusoidal function as its value is bounded like a sinusoid.

Now, if the output function used is ReLu then the output is unbounded. But still it is just some linear combination of weights gone through some non-linear functions (in the previous layers which at best might be linearly unbounded if previous layers are ReLu). So one can assume, that even though the Neural Net can approximate an unbounded linear function, can it approximate an unbounded Polynomial function or an exponential function?

$\star$ Although the regression problem might seem more suitable to Fourier Transform analogy than Fourier Series, I have used the FS analogy based on the fact that FT output is is continuous function as opposed to FS (in NN regression we are adding outputs of several nodes, similar to what we do in FS where $number_{nodes} << \infty$.

$\endgroup$
  • $\begingroup$ Any proof of this "since the best approximation of a function the Fourier Series demands Dirichlet's condition to be satisfied"? Can you also be more precise? What do you mean by "best"? $\endgroup$ – nbro May 10 at 12:36
  • $\begingroup$ "Now, if the output function used is ReLu then the output is unbounded", I am also not sure this is always the case. It is unbounded only if the input is unbounded (recall the definition of ReLU). Also, what is the relation between this previous statement and "But still it is just some linear combination of weights gone through some non-linear functions (the previous layers)"? $\endgroup$ – nbro May 10 at 12:42
  • $\begingroup$ I think you should simplify your post: only one question. You're asking too many specific questions. I would just ask "Can a Neural Network do regression for an unbounded function?" and I would not provide any dubious assumptions. $\endgroup$ – nbro May 10 at 12:48
  • $\begingroup$ @nbro no dubious assumptions provided. A best approximation of a function can only be done by Fourier approximation as a sum of sinusoidal functions. It's the basis of digital communication and signal processing. By relu unbounded I meant it's the only conventional activation function with an unbounded output. It means that the previous layers assuming non relu are still bounded, and thus some linear combination of weights through some non linear transformation. $\endgroup$ – DuttaA May 10 at 15:33
  • $\begingroup$ What do you mean by best? Fourier series is a way of approximating a function, yes, but what do you mean by "best"? $\endgroup$ – nbro May 10 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.