1
$\begingroup$

The following image is a screenshot from a video tutorial that illustrates the concept of gradient descent algorithm with a 3D animation.

Do the numbers on the top of the balls pointed out by the red arrows represent the gradient?

enter image description here

$\endgroup$
9
  • $\begingroup$ When you say "Are the numbers on the top of the balls are their gradient descent?", do you mean "gradient" (and not "gradient descent"), right? I think so, thus I edited your post to reflect this. Also, note that you were watching a video that has already 2 dislikes, so that may be a good sign that you shouldn't even watch that video. $\endgroup$ – nbro Apr 10 '20 at 22:48
  • $\begingroup$ @nbro With your reminder I realize that, thank you. Could you explain a bit more about the mistake in that video? For example, gradient usually ranges between (-1.0, 1.0), is my understanding right? $\endgroup$ – Piete3r Apr 11 '20 at 0:16
  • $\begingroup$ I am not saying that the video is showing the wrong info. The gradient is a vector and the values of the elements of this gradient (i.e. a vector) aren't really restricted to a range. The values depend on the loss function. I think you should investigate a little bit more about derivatives, gradients, etc., then ask a more specific question. $\endgroup$ – nbro Apr 11 '20 at 0:18
  • $\begingroup$ @nbro Thank you. My OP is asking a specific question and your comment to that question is appreciated. I guess I understand the basic idea about derivatives, gradients in general. The range of (-1.0, 1.0) I use is specifically for the discussion in the context of deep learning, where all features are typically normalized. Please correct my misunderstanding. Sorry for misleading. $\endgroup$ – Piete3r Apr 11 '20 at 0:41
  • $\begingroup$ Well, yes, typically, you don't want the gradient to take very big or very small numbers, but this not guaranteed, and, as you say, it may also depend on the activation functions that you use. Have a look at the exploding and vanishing gradient problems. $\endgroup$ – nbro Apr 11 '20 at 0:42
1
$\begingroup$

It represents the value of the loss function J(x1, x2; θ); the valley has value 0 in the video. You can see that the lowest ball with value 3.13 is on a steep point with a high gradient, so it's not the gradient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.