Neural Network is trained to learn a non-linear function, the more layers it has, the more is the quality of the prediction and the ability to match the real-world function correctly (lets leave aside overfitting for now).

Now, given that the concept of a derivative of a function (if it is not a line) is only defined at a single point , and worse than that, it is defined as a tangent line (so, its linear in essence) , and backpropagation uses this tangent line to project weight updates, does this mean that Gradient Descent is linear at its fundamental level ?

Then if Gradient Descent is using linearity to propagate back weight updates, imagine how big the error would be for non linear function? For example , you have calculated that the weight number 129 of your network has to be decreased by 0.003412, but at that point with this new weight, the function might have already reversed its direction, and the real update for this weight must be a negative number!!! Isn't this the reason that our deep fully connected networks have such a difficulty to learn , because with more layers stacked up, the more non-linear the model becomes, and thus, the weight updates we propagate back to lower layers could be treated as "best guess" values instead of something that can be fully trusted?

I am correct in assuming that Gradient Descent is not calculating correct weight updates on each backwards step and that the only reason that the network eventually converges to required model is because these imprecisions are fixed in a loop (called epoch). So, if we use an analogy, Gradient Descent would be like navigating the World on a boat with maps developed with the assumption that the Earth is flat. With such maps, you can sail in near-by areas, but if you would travel around the world without knowing that the Earth is round you will never arrive to your destination, and that's exactly what we are experiencing when we train deep fully connected networks without making them converge.

This means, if Gradient Descent is broken in such a way, a correct Gradient Descent algorithm would only have to do SINGLE backwards step and update all the weights in one pass, giving the minimum error that is theoretically possible in 1 epoch .... I am right ?

So, my question basically is: is Gradient Descent a really broken algorithm or I am missing something?

  • $\begingroup$ What? Very confusing question to me. Are you saying that by the time you reach weight 129 after updating all weights till 128 the function value has already changed, or are you saying that the derivative changes value due to non linearity as it propagates backwards? Or you are saying none of the above. $\endgroup$
    – user9947
    Apr 9 '20 at 16:05
  • $\begingroup$ @DuttaA, well there might be a condition that after a change in this single weight (w129) by 0.003412, the Error will be totally different, and the function at that interval might be concave up now (after the change), while before this weight update gradient descent was assuming that the function is concave down. So, the weight update was incorrect $\endgroup$
    – Nulik
    Apr 9 '20 at 16:32
  • $\begingroup$ I mean we adjust weights to change the error, so the error will change and also the error surface will change (due to effect of other weights). Can't really understand your question though. $\endgroup$
    – user9947
    Apr 10 '20 at 0:06
  • $\begingroup$ @DuttaA the question is easy: is it correct to change weights using a linear function when dealing with non-linear function? the real world function is non-linear , and it may not even be continous, and not differentiable at all $\endgroup$
    – Nulik
    Apr 10 '20 at 0:21
  • $\begingroup$ I mean second order derivatives exist which probably takes care of this, the learning rate is also responsible. $\endgroup$
    – user9947
    Apr 10 '20 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.