# Effects of ReLU Activation on Convexity of Loss Functions

I have heard the following argument being made regarding Neural Networks:

• A Neural Network is a composition of several Activation Functions
• Sigmoid Activation Functions are Non-Convex Functions
• The composition of Non-Convex Functions can produce a Non-Convex Function
• Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):

e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")


My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

• On it's own, the ReLU function is said to be Convex.
• Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.

However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.

Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?

Thanks!

• References:

Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.

• single layer with relu activation should be convex. But compositions of convex functions are not necessarily convex, so even in the case of a single layer with relu, the loss function may not be convex.
– Taw
Jan 24, 2022 at 15:24
• Also, you should note that relu is a nonlinear function - just like sigmoid.
– Taw
Jan 24, 2022 at 15:26
• @ Taw: thank you for your reply! what do you mean by "compositions"? do you mean "a loss function composed of several relu functions"? Jan 24, 2022 at 22:20
• @ Taw: can you please explain how "Relu is a non-linear function"? it seems to be a piecewise linear function, no? Jan 24, 2022 at 22:20
• @stats555 piecewise linear functions are not linear, hence they are nonlinear. A composition of two functions $f, y$ is defined as $f(y(x))$ where $x$ is input. E.g. call the model y, call the loss function $f$. Then the loss function used in deep learning would look something like $f(\hat y, y(x))$ where $\hat y$ is the target, $y$ is the model, $x$ is the input. My point is, even if $y$ is a single layer with relu, $f(\hat y, y(x))$ is not necessarily convex.
– Taw
Jan 25, 2022 at 5:26

You're missing a couple of quite important concepts:

• Universal approximation theorem: with enough parameters a neural network can approximate any function.
• Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )

But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like $$sin(x)$$ with relu:

from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

f = lambda x: [[x_] for x_ in x]
noise_level = 0.1
X_train_ = np.arange(0, 10, 0.2)
real_sin = np.sin(X_train_)
y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
nodes = 1000
layers = 10
regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation="relu").fit(f(X_train_), y_train)
predicted_sin = regr.predict(f(X_train_))

plt.plot(X_train_, real_sin, label="sin target")
plt.plot(X_train_, predicted_sin, label="sin predicted")
plt.legend()
plt.show()


You'll see it's not a task too hard to learn:

PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.

• I would disagree, that every loss function is non-convex. Mean squared error and Crossentropy (Binary or Multilabel) are convex functions. Jan 29, 2022 at 6:21
• Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread math.stackexchange.com/questions/2402455/… Jan 29, 2022 at 16:54
• one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex. Jan 29, 2022 at 18:37

I think you are asking about Fully Input Convex Neural Networks as proposed in [1]. ReLU is in fact a convex function, and the sum of convex functions can only produce convex functions. However, unlike you said, composition of convex functions can produce non-convex functions, unless they are non-decreasing. With FICNNs you can only learn convex functions. For that all weights W must be non-negative for the activation function g

From [1] the interesting part is:

The function f is convex in y provided that all $$W_i(z)_{1:k}$$ are non-negative, and all functions $$g_i$$ are convex and non-decreasing. The proof is simple and follows from the fact that non-negative sums of convex functions are also convex and that the composition of a convex and convex non-decreasing function is also convex (see e.g. Boyd & Vandenberghe (2004, 3.2.4)).

[1] Amos, Brandon, Lei Xu, and J. Zico Kolter. "Input convex neural networks." International Conference on Machine Learning. PMLR, 2017.

• I'd recommend using LaTeX to format your equations and symbols for example $W_i$ and $g_i$ Nov 16, 2022 at 14:59