As far as my knowledge goes (might be a bit vague and not mathematical), a Neural Network can and should only be able to approximate a bounded function, which is not the case of a Polynomial Regressor. For instance, the target function is

$$f_t(x_i) = a_2x_1^3 + a_1x_2^3 + a_0$$

In a Polynomial Regressor, we take a combination function of inputs. The hypothesised function might look like:

$$f_h(x_i) = b_6x_1^3 + b_5x_2^3 + b_4x_1^2x_2 + b_3x_1x_2^2 + b_2x_1x_2 + b_1x_1 + b_0 $$

with a dataset of observations.

So, the best way to approximate $f_t$ with $f_h$ is to set all the coefficients of $f_h$ to zero, except for $b_6, b_5, b_0$, which should be equal to the corresponding coefficients in $f_t$, i.e. $b_6 = a_2$, $b_5 = a_1$ and $b_0 = a_0$ (one can easily show this by letting $f_h(x) = f_t(x) \quad, \forall x$).

The case of a Neural Network Regressor is as follows. The output of the final layer of a Neural Network is taken, multiplied with a set of weights, and summed up, which becomes our hypothesized functions. So, $$f_{NN}(x_i) = f_l(x_i) = \sum_{i=0}^{N} w_if_{l-1}^i(x_i),$$ where $N$ is the number of nodes in the last layer, $l$ denotes the layer and $w_i$ weight for a node. We use a ReLU activation (since it is the most widely used).

So, after learning the model, if we want to hypothesize, what is the maximum output value the Neural Network can reach? This will be clearly the case when the input to all nodes of the Neural Network lies in the positive region, hence linear and hence can be written as:

$$f_{NN}(x_i) =X^T W$$

Clearly, a linear function. So, can it approximate a cubic polynomial, which is our real model? Clearly, NO (can be proven by L'Hospitals rule), since a polynomially growing function cannot be approximated by a linearly growing function. Can it approximate within the given range of the dataset? Clearly YES (if enough depth and nodes are provided)

So, basically in my understanding, Neural Networks can only approximate BIBO (Bounded Input, Bounded Output) functions.

So, we see the following disadvantages of a Neural Network Regressor w.r.t a Polynomial Regressor:

  • It cannot approximate an unbounded function (with unbounded inputs).
  • The output will be aliased to some degree (depending on the size of the NN) as a somewhat (not strictly) linear function is approximating a non-linear function.

My questions are:

  1. Is my previous reasoning wrong? If so, how?

  2. If my reasoning is correct, what are the steps taken/should be taken so that a Neural Network Regressor can eradicate these drawbacks against a Polynomial Regressor?

  • $\begingroup$ relu is piecewise linear, so its not actually linear, so you cannot write a NN as a linear model $\endgroup$
    – mshlis
    Aug 12 '19 at 2:30
  • $\begingroup$ @mshlis I wrote 'not strictly' $\endgroup$
    – user9947
    Aug 12 '19 at 3:25
  • 1
    $\begingroup$ @DuttaA I'm not entirely sure what you're asking, so sorry if this is irrelevant (I just skim read and a lot of notation takes a while for me to get my head around) but what you said here: " So can it approximate a cubic polynomial, which is our real model? Clearly NO" doesn't sound right. A relu can very easily approximate polynomials, see here: desmos.com/calculator/cfvtjusqmq for a really nice example. Also, in the case that it doesn't, what's stopping you using a different activation function, say a sigmoid? $\endgroup$
    – Recessive
    Aug 12 '19 at 4:20
  • $\begingroup$ @Recessive so a function growing cubically can be approximated by a function which can at most grow linearly (considering finite size of NN)? Also I can walk you through the parts you didn't understand since it's a big question I might have inadvertently missed somethings which might not be obvious. As far as sigmoid goes it will be even worse. $\endgroup$
    – user9947
    Aug 12 '19 at 4:42
  • $\begingroup$ @DuttaA Ohh I think I see what you're saying. I would agree with you then, no a neural network (with relu, and i think would still struggle with something else) couldn't compete on this front, but I think that's mostly because neural networks aren't made with the intention of dealing with all possible inputs. I guess a good analogy would be a car/truck identifier. This classifier is not built with the capacity to handle an input such as complete random noise. In this case, an approximator for a function such as this is designed to work in a range, not from -inf to +inf. $\endgroup$
    – Recessive
    Aug 12 '19 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy