Neural Network is trained to learn a non-linear function, the more layers it has, the more is the quality of the prediction and the ability to match the real-world function correctly (lets leave aside overfitting for now).
Now, given that the concept of a derivative of a function (if it is not a line) is only defined at a single point , and worse than that, it is defined as a tangent line (so, its linear in essence) , and backpropagation uses this tangent line to project weight updates, does this mean that Gradient Descent is linear at its fundamental level ?
Then if Gradient Descent is using linearity to propagate back weight updates, imagine how big the error would be for non linear function? For example , you have calculated that the weight number 129 of your network has to be decreased by 0.003412, but at that point with this new weight, the function might have already reversed its direction, and the real update for this weight must be a negative number!!! Isn't this the reason that our deep fully connected networks have such a difficulty to learn , because with more layers stacked up, the more non-linear the model becomes, and thus, the weight updates we propagate back to lower layers could be treated as "best guess" values instead of something that can be fully trusted?
I am correct in assuming that Gradient Descent is not calculating correct weight updates on each backwards step and that the only reason that the network eventually converges to required model is because these imprecisions are fixed in a loop (called epoch). So, if we use an analogy, Gradient Descent would be like navigating the World on a boat with maps developed with the assumption that the Earth is flat. With such maps, you can sail in near-by areas, but if you would travel around the world without knowing that the Earth is round you will never arrive to your destination, and that's exactly what we are experiencing when we train deep fully connected networks without making them converge.
This means, if Gradient Descent is broken in such a way, a correct Gradient Descent algorithm would only have to do SINGLE backwards step and update all the weights in one pass, giving the minimum error that is theoretically possible in 1 epoch .... I am right ?
So, my question basically is: is Gradient Descent a really broken algorithm or I am missing something?