Effects of ReLU Activation on Convexity of Loss Functions

Question

I have heard the following argument being made regarding Neural Networks:

• A Neural Network is a composition of several Activation Functions
• Sigmoid Activation Functions are Non-Convex Functions
• The composition of Non-Convex Functions can produce a Non-Convex Function
• Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):

e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")

My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

• On it's own, the ReLU function is said to be Convex.
• Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.

However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.

Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?

Thanks!

• References:

https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html

Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.

Answer 1

You're missing a couple of quite important concepts:

• Universal approximation theorem: with enough parameters a neural network can approximate any function.
• Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )

But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like $sin(x)$ with relu:

from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

f = lambda x: [[x_] for x_ in x]
noise_level = 0.1
X_train_ = np.arange(0, 10, 0.2)
real_sin = np.sin(X_train_)
y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
nodes = 1000
layers = 10
regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation="relu").fit(f(X_train_), y_train)
predicted_sin = regr.predict(f(X_train_))

plt.plot(X_train_, real_sin, label="sin target")
plt.plot(X_train_, predicted_sin, label="sin predicted")
plt.legend()
plt.show()

You'll see it's not a task too hard to learn:

PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.

Answer 2

I think you are asking about Fully Input Convex Neural Networks as proposed in [1]. ReLU is in fact a convex function, and the sum of convex functions can only produce convex functions. However, unlike you said, composition of convex functions can produce non-convex functions, unless they are non-decreasing. With FICNNs you can only learn convex functions. For that all weights W must be non-negative for the activation function g

From [1] the interesting part is:

The function f is convex in y provided that all $W_i(z)_{1:k}$ are non-negative, and all functions $g_i$ are convex and non-decreasing. The proof is simple and follows from the fact that non-negative sums of convex functions are also convex and that the composition of a convex and convex non-decreasing function is also convex (see e.g. Boyd & Vandenberghe (2004, 3.2.4)).

[1] Amos, Brandon, Lei Xu, and J. Zico Kolter. "Input convex neural networks." International Conference on Machine Learning. PMLR, 2017.