Do we know the Effect of "RELU Activation Functions" on the Convexity of the Loss Functions in Neural Networks?

I have heard the following argument being made regarding Neural Networks:

• Sigmoid Activation Functions are Non-Convex Functions
• Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions are Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):

e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")

My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

"RELU Activation Functions" are said to solve certain problems that occur (e.g. vanishing gradient, exploding gradient) when training Neural Networks:

However, the discontinuities and the differentiability of the "RELU Activation Function" make it unclear to me whether or not Loss Functions that contain "RELU Activation Functions" would automatically be Convex or Non-Convex.

Can someone please comment on this? Due to the behavior of RELU Activation Functions- do Neural Networks with RELU Activation Functions "automatically" become Non-Convex?

Thanks!

• References:

https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html

Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.

I have heard the following argument being made regarding Neural Networks:

• A Neural Network is a composition of several Activation Functions
• Sigmoid Activation Functions are Non-Convex Functions
• The composition of Non-Convex Functions can produce a Non-Convex Function
• Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):

e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")

My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

• On it's own, the ReLU function is said to be Convex.
• Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.

However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.

Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?

Thanks!

• References:

https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html

Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.