# What is the derivative function used in backpropagration?

I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function?

These terms are confusing: derivative of act. function, partial derivative wrt. loss function??

I'm still not getting it correct.

Overview

The derivatives of functions are used to determine what changes to input parameters correspond to what desired change in output for any given point in the forward propagation and cost, loss, or error evaluation &mdash whatever it is conceptually the learning process is attempting to minimize. This is the conceptual and algebraic inverse of maximizing valuation, yield, or accuracy.

Back-propagation estimates the next best step toward the objective quantified in the cost function in a search. The result of the search is a set of parameter matrices, each element of which represents what is sometimes called a connection weight. The improvement of the values of the elements in the pursuit of minimal cost is artificial networking's basic approach to learning.

Each step is an estimation because the cost function is a finite difference, where as the partial derivatives express the slope of a hyper-plane normal to surfaces that represent functions that comprise forward propagation. The goal is to set up circumstances so that successive approximations approach the ideal represented by minimization of the cost function.

Back-propagation Theory

Back-propagation is a scheme for distribution of a correction signal arising from cost evaluation after each sample or mini-batch of them. With a form of Einsteinian notation, the current convention for distributive, incremental parameter improvement can be expressed concisely.

$$\Delta P = \dfrac {c(\vec{o}, \vec{\ell}) \; \alpha} {\big[ \prod^+ \! P \big] \; \big[ \prod^+ \!a'(\vec{s} \, P + \vec{z}) \big] \; \big[ c'(\vec{o}, \vec{\ell}) \big]}$$

The plus sign in $$\prod^+\!$$ designates that the factors multiplied must be downstream in the forward signal flow from the parameter matrix being updated.

In sentence form, $$\Delta P$$ at any layer shall be the quotient of cost function $$c$$ (given label vector $$\vec{\ell}$$ and network output signal $$\vec{o}$$), attenuated by learning rate $$\alpha$$, over the product of all the derivatives leading up to the cost evaluation. The multiplication of these derivatives arise through the recursive application of the chain rule.

It is because the chain rule is a core method for feedback signal evaluation that partial derivatives must be used. All variables must be bound except for one dependent and one independent variable for the chain rule to apply.

The derivatives include three types.

• All layer input factors, the weights in the parameter matrix used to attenuate the signal during forward propagation, which are equal to the derivatives of those signal paths
• All the derivatives of activation functions $$a$$ evaluated at the sum of the matrix vector product of the parameters and signal at that layer plus the bias vector
• The derivative of the cost function $$c$$ evaluated at the current output value $$\vec{o}$$ with the label $$\vec{\ell}$$