0
$\begingroup$

From https://stackoverflow.com/questions/36370129/does-tensorflow-use-automatic-or-symbolic-gradients, I understood TensorFlow requires all the operations in the Graph to be explicit formulas (instead of black-boxes, such as raw python functions) to do Automatic Differentiation. Then it will do some kind of Gradient Descent based on that to minimization.

I'm wondering, since it already know all the explicit formulas, can it directly find out the minimum by examining the equation itself? Like computing the points where gradient is zero or do not exist, then do some kind of processing to find out the minimum.

I found it is simple to do this "symbolic minimization" above with few variables such as minimizing Σ(a_i - v)^2 where v is the trainable variable an a_i are all the training samples. I'm not sure is there a general way though.

$\endgroup$
0
$\begingroup$

If by "symbolic" you mean finding an analytical solution, that is, an equation for each weight, then the answer is no. The example you chose results in a system linear equations, which can be solved analytically. However once you introduce non linearities (by using activation functions with more than one layer), most non trivial cases will have no analytical solution and will need to be solved numerically. This is not a problem specific to tensorflow, it is a mathematical issue, it will not be possible on any language, current or future. Unless there is some revolution in math first.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.