Which class of functions are quite complicated in deep learning?

Question

Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.

Activation functions are a part of neural network function and neural network functions may be a part of loss functions.

Consider the following paragraphs from Numerical Computation of a deep learning book

Optimization algorithms that use only the gradient, such as gradient descent, are called first-order optimization algorithms. Optimization algorithms that also use the Hessian matrix, such as Newton’s method, are called second-order optimization algorithms (Nocedal and Wright, 2006).

The optimization algorithms employed in most contexts in this book are applicable to a wide variety of functions but come with almost no guarantees. Deep learning algorithms tend to lack guarantees because the family of functions used in deep learning is quite complicated. In many other fields, the dominant approach to optimization is to design optimization algorithms for a limited family of functions.

The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?

Answer 1

From the phrasing, it seems that complicated refers to the non-convexity of the loss landscapes of neural networks. We do not have formal guarantees of convergence in general for such landscapes. This non-convexity is a property of both the function defined by the neural network, and the particular loss function we use.

In practice though, non-convexity stems from the non-linear activation functions as we almost exclusively use cross-entropy loss when training neural networks.