Today's computers can calculate symbolic derivatives quite fast, why is automatic differentiation still used? For example, Mathematica can handle algebraic operations with arrays. Doesn't automatic differentiation cause significant overhead? Calculating the symbolic gradient of an MLP should not be too difficult, or am I wrong?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
That depends on the point-of-view. If you have some function given as a string of its mathematical formula and the output is the fully inserted string of its derivative, then the chain rule $(u\circ v)'(x)=u'(v(x))v'(x)$ quasi duplicates the occurrence of $v$. This happens for each composition, so one can get an exponential growth in the string length from input to output.
If you keep common sub-expressions separate, then what you do is parsing the function into a computational graph and taking the derivative of it. But this is already in the territory of algorithmic/automatic differentiation. If you do not keep the derivatives inside the original nodes, but split off additional nodes, then the output graph will double to triple in size relative to the original graph. This is a very compact representation of the symbolic derivative. Such structures have long been used and sometimes also exposed in CAS systems like Maple.
The aim of AD is the fast evaluation of derivatives, so if evaluating in a point in a first sweep the node values and derivatives relative to the node input are computed, and each additional forward or backward sweep is just linear operations to compute tangents or gradients.
In a ANN the network itself represents the computational graph, matrix-vector multiplication nodes/layers have the matrix and the input vector as the required node derivatives, for the scalar values of the activation function the node derivative needs to be computed separately. As the gradient computation has only one backward pass for each input, they can be computed just when they are needed.
In short, backprop networks are already a part or an application of AD, there is not really a way to make this more efficient. In some cases (sparse graphs, multiple outputs,...) it can be useful to apply AD techniques to compress parts of the network, if the organizational overhead justifies the gains.
