# Effects of ReLU Activation on Convexity of Loss Functions

I have heard the following argument being made regarding Neural Networks:

• A Neural Network is a composition of several Activation Functions
• Sigmoid Activation Functions are Non-Convex Functions
• The composition of Non-Convex Functions can produce a Non-Convex Function
• Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):

e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")


My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

• On it's own, the ReLU function is said to be Convex.
• Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.

However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.

Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?

Thanks!

• References:

Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.

• single layer with relu activation should be convex. But compositions of convex functions are not necessarily convex, so even in the case of a single layer with relu, the loss function may not be convex.
– Taw
Jan 24 at 15:24
• Also, you should note that relu is a nonlinear function - just like sigmoid.
– Taw
Jan 24 at 15:26
• @ Taw: thank you for your reply! what do you mean by "compositions"? do you mean "a loss function composed of several relu functions"? Jan 24 at 22:20
• @ Taw: can you please explain how "Relu is a non-linear function"? it seems to be a piecewise linear function, no? Jan 24 at 22:20
• @stats555 piecewise linear functions are not linear, hence they are nonlinear. A composition of two functions $f, y$ is defined as $f(y(x))$ where $x$ is input. E.g. call the model y, call the loss function $f$. Then the loss function used in deep learning would look something like $f(\hat y, y(x))$ where $\hat y$ is the target, $y$ is the model, $x$ is the input. My point is, even if $y$ is a single layer with relu, $f(\hat y, y(x))$ is not necessarily convex.
– Taw
Jan 25 at 5:26

You're missing a couple of quite important concepts:

• Universal approximation theorem: with enough parameters a neural network can approximate any function.
• Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )

But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like $$sin(x)$$ with relu:

from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

f = lambda x: [[x_] for x_ in x]
noise_level = 0.1
X_train_ = np.arange(0, 10, 0.2)
real_sin = np.sin(X_train_)
y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
nodes = 1000
layers = 10
regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation="relu").fit(f(X_train_), y_train)
predicted_sin = regr.predict(f(X_train_))

plt.plot(X_train_, real_sin, label="sin target")
plt.plot(X_train_, predicted_sin, label="sin predicted")
plt.legend()
plt.show()


You'll see it's not a task too hard to learn:

PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.

• I would disagree, that every loss function is non-convex. Mean squared error and Crossentropy (Binary or Multilabel) are convex functions. Jan 29 at 6:21
• Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread math.stackexchange.com/questions/2402455/… Jan 29 at 16:54
• one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex. Jan 29 at 18:37