So as far as my knowledge (might be a bit vague and not mathematical) goes a Neural Network can and should only be able to approximate a bounded function, which is not the case of a Polynomial Regressor. Say for instance the target function is. $$f_t(x_i) = a_2x_1^3 + a_1x_2^3 + a_0$$

Now in a Polynomial Regressor we take a combination function of inputs, like for example the hypothesised function might look like: $$f_h(x_i) = b_6x_1^3 + b_5x_2^3 + b_4x_1^2x_2 + b_3x_1x_2^2 + b_2x_1x_2 + b_1x_1 + b_0 $$ with a dataset of observations within a give range (all numbers are real): $$r^1_1 < x_1 < r^1_2$$ $$r^2_1 < x_2 < r^2_2$$

So, if we do everything correctly and assuming ideal case except $b_6, b_5, b_0$ all the co-efficients should tend to $0$ while the non-zero co-efficients should tend to the value in the original model we are trying to approximate.

Now, the case of a Neural Network Regressor is as follows, the output of the final layer of a Neural Network is taken, multiplied with a set of weights and summed up which becomes our hypothesised functions. So, $$f_{NN}(x_i) = f_l(x_i) = \sum_{i=0}^{N} w_if_{l-1}^i(x_i)$$ where $N$ is the number of nodes in the last layer, $l$ denotes the layer and $w_i$ weight for a node. We use a ReLu activation (since it is the most widely used).

So after learning the model if we want to hypothesize what is the maximum value the Neural Network can reach? This will be clearly the case when input to all nodes of the Neural Network lie in the positive region, hence linear and hence can be written as:

$$f_{NN}(x_i) =X.W^T$$

Clearly a linear function. So can it approximate a cubic polynomial, which is our real model? Clearly NO (can be proven by L'Hospitals rule), since a polynomially growing function cannot be approximated by a linearly growing function. Can it approximate within the given range of dataset? Clearly YES (if enough depth and nodes are provided)

So basically in my understanding Neural Networks can only approximate BIBO (Bounded Input, Bounded Output) functions.

So we see the following disadvantages of a Neural Network Regressor w.r.t a Polynomial Regressor:

  • It cannot approximate an unbounded function (with unbounded inputs).
  • The output will be aliased to some degree (depending on the size of the NN) as a somewhat (not strictly) linear function is approximating a non-linear function.

So my questions are:

  • Is my previous reasoning wrong? If so how?
  • If my reasoning is correct what are the steps taken/should be taken so that a Neural Network Regressor can eradicate these drawbacks against a Polynomial Regressor.

SIDE NOTE: In most study resources they are always sugar coating and bloating about Neural Networks in comparison to regressors, but this seems to me a flaw which is quite significant.

  • $\begingroup$ relu is piecewise linear, so its not actually linear, so you cannot write a NN as a linear model $\endgroup$ – mshlis Aug 12 '19 at 2:30
  • $\begingroup$ @mshlis I wrote 'not strictly' $\endgroup$ – DuttaA Aug 12 '19 at 3:25
  • $\begingroup$ @DuttaA I'm not entirely sure what you're asking, so sorry if this is irrelevant (I just skim read and a lot of notation takes a while for me to get my head around) but what you said here: " So can it approximate a cubic polynomial, which is our real model? Clearly NO" doesn't sound right. A relu can very easily approximate polynomials, see here: desmos.com/calculator/cfvtjusqmq for a really nice example. Also, in the case that it doesn't, what's stopping you using a different activation function, say a sigmoid? $\endgroup$ – Recessive Aug 12 '19 at 4:20
  • $\begingroup$ @Recessive so a function growing cubically can be approximated by a function which can at most grow linearly (considering finite size of NN)? Also I can walk you through the parts you didn't understand since it's a big question I might have inadvertently missed somethings which might not be obvious. As far as sigmoid goes it will be even worse. $\endgroup$ – DuttaA Aug 12 '19 at 4:42
  • $\begingroup$ @DuttaA Ohh I think I see what you're saying. I would agree with you then, no a neural network (with relu, and i think would still struggle with something else) couldn't compete on this front, but I think that's mostly because neural networks aren't made with the intention of dealing with all possible inputs. I guess a good analogy would be a car/truck identifier. This classifier is not built with the capacity to handle an input such as complete random noise. In this case, an approximator for a function such as this is designed to work in a range, not from -inf to +inf. $\endgroup$ – Recessive Aug 12 '19 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.