3
$\begingroup$

Training on a quadratic function

x = np.linspace(-10, 10, num=1000)
np.random.shuffle(x)
y = x**2

Will predict an expected quadratic curve between -10 < x < 10.

in bound

Unfortunately my model's predictions become linear outside of the trained dataset.

See -100 < x < 100 below:

out of bounds

Here is how I define my model:

model = keras.Sequential([
      layers.Dense(64, activation='relu'),
      layers.Dense(64, activation='relu'),
      layers.Dense(1)
  ])

model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.1))

history = model.fit(
    x, y,
    validation_split=0.2,
    verbose=0, epochs=100)

Here's a link to a google colab for more context.

$\endgroup$
2
$\begingroup$

It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.

Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.

Proposition. In a single-layer neural network with $n$ hidden neurons using $\mathrm{ReLU}$ activation, with one input and output node, the output is linear outside of the region $[A, B]$ for some $A < B \in \mathbb{R}$. In other words, if $x > B$, $f(x) = \alpha x + \beta$ for some constants $\alpha$ and $\beta$, and if $x < A$, $f(x) = \gamma x + \delta$ for some constants $\gamma$ and $\delta$.

Proof. I can write the neural network as a function $f \colon \mathbb R \to \mathbb R$, defined by $$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$ Note that each neuron switches from being $0$ to a linear function, or vice versa, when $w_i x + b_i = 0$. Define $r_i = -\frac{b_i}{w_i}$. Then, I can set $B = \max_i r_i$ and $A = \min_i r_i$. If $x > B$, each neuron will either be $0$ or linear, so $f$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $x < A$.

Hence, $f$ is a linear function with constant gradient if $x < A$ or $x > B$. $\square$

If the result isn't clear, here's an illustration of the idea: Plot of sum of ReLU

This is a $3$-neuron network, and I've marked the points I denote $r_i$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.

$\endgroup$
2
  • $\begingroup$ Am I misunderstanding this, or do different activation functions lead to the same result because they just activate the y = weight*x + bias linear function? $\endgroup$ – Mr. Demetrius Michael Apr 9 at 14:35
  • $\begingroup$ Well, you'd get slightly different results with other activations. As an example take a layer of neurons with sigmoid activation: you'd essentially have a function which is a sum of (scaled and repositioned) sigmoids. I haven't thought about it very deeply but I expect that as $x \to \infty$ you'd see each neuron saturate (either at zero or one) so the function would be "almost constant" after a certain point. Play around on a graph plotter with sums of logistic functions to see what I mean, it's a fun exercise. $\endgroup$ – htl Apr 9 at 14:59
1
$\begingroup$

Short answer: Yes.

Consider a non-linear regression on that dataset. Using a model of degree two, it would fit a quadratic exactly to your perfect data here. But I suppose you're asking about neural networks. You can have neural networks set up that are exactly equivalent to this kind of regression, so even with neural networks, yes you can get this non-linear extrapolation. Of course as you probably realise, you would have to know in advance what kind of behaviour you expect in this extrapolation before really trusting any extrapolated predictions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.