Deep learning is a field in which we need neural networks that are deep enough to carry on our task. The important fucntions in deep neural networks can be classified in to three classes: activation function, neural network function and loss function.
Activation functions are a part of neural network function and neural network functions may be a part of loss functions.
Consider the following paragraphs from Numerical Computation of a deep learning book
Optimization algorithms that use only the gradient, such as gradient descent, are called ﬁrst-order optimization algorithms. Optimization algorithms that also use the Hessian matrix, such as Newton’s method, are called second-order optimization algorithms (Nocedal and Wright, 2006).
The optimization algorithms employed in most contexts in this book are applicable to a wide variety of functions but come with almost no guarantees. Deep learning algorithms tend to lack guarantees because the family of functions used in deep learning is quite complicated. In many other ﬁelds, the dominant approach to optimization is to design optimization algorithms for a limited family of functions.
The last passage is talking about the family of functions used in deep learning. Which class of functions, among the three I mentioned, they are referring to?