# Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?

x = np.linspace(-10, 10, num=1000)
np.random.shuffle(x)
y = x**2


Will predict an expected quadratic curve between -10 < x < 10.

Unfortunately my model's predictions become linear outside of the trained dataset.

See -100 < x < 100 below:

Here is how I define my model:

model = keras.Sequential([
layers.Dense(64, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(1)
])

history = model.fit(
x, y,
validation_split=0.2,
verbose=0, epochs=100)


It isn't too surprising to see behaviour like this, since you're using $$\mathrm{ReLU}$$ activation.

Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.

Proposition. In a single-layer neural network with $$n$$ hidden neurons using $$\mathrm{ReLU}$$ activation, with one input and output node, the output is linear outside of the region $$[A, B]$$ for some $$A < B \in \mathbb{R}$$. In other words, if $$x > B$$, $$f(x) = \alpha x + \beta$$ for some constants $$\alpha$$ and $$\beta$$, and if $$x < A$$, $$f(x) = \gamma x + \delta$$ for some constants $$\gamma$$ and $$\delta$$.

Proof. I can write the neural network as a function $$f \colon \mathbb R \to \mathbb R$$, defined by $$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$ Note that each neuron switches from being $$0$$ to a linear function, or vice versa, when $$w_i x + b_i = 0$$. Define $$r_i = -\frac{b_i}{w_i}$$. Then, I can set $$B = \max_i r_i$$ and $$A = \min_i r_i$$. If $$x > B$$, each neuron will either be $$0$$ or linear, so $$f$$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $$x < A$$.

Hence, $$f$$ is a linear function with constant gradient if $$x < A$$ or $$x > B$$. $$\square$$

If the result isn't clear, here's an illustration of the idea:

This is a $$3$$-neuron network, and I've marked the points I denote $$r_i$$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.

• Am I misunderstanding this, or do different activation functions lead to the same result because they just activate the y = weight*x + bias linear function? – Mr. Demetrius Michael Apr 9 at 14:35
• Well, you'd get slightly different results with other activations. As an example take a layer of neurons with sigmoid activation: you'd essentially have a function which is a sum of (scaled and repositioned) sigmoids. I haven't thought about it very deeply but I expect that as $x \to \infty$ you'd see each neuron saturate (either at zero or one) so the function would be "almost constant" after a certain point. Play around on a graph plotter with sums of logistic functions to see what I mean, it's a fun exercise. – htl Apr 9 at 14:59