0
$\begingroup$

Consider the following definition of derivative from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.

Definition 5.2 (Derivative). More formally, for $h>0$ the derivative of $f$ derivative at $x$ is defined as the limit

$$\dfrac{df}{dx} := \lim\limits_{h \rightarrow 0}^{} \dfrac{f(x + h) − f(x)}{h}$$

The derivative of $f$ points in the direction of the steepest ascent of $f$.

You can observe that the derivate of a function is another function. If we consider derivative at a single point then it will be a real number that quantifies the rate of change of the output of the function with respect to the input.

There are two kinds of directions we need to focus on that are related to gradients. One is the direction pointed by a gradient and another one is the direction for moving our input parameters using a gradient. This question is restricted to the direction of the first kind.

We can treat the sign of the derivative at a particular point as the direction to move our input parameters. And I am not sure about the rigorous definition for the direction pointed by a derivative. I have thus doubts about the direction pointed by a gradient.

What exactly is the direction pointed by a gradient? and I want to know the formal definition for the direction of a gradient

I know about the direction that is given by gradient to move our parameters. But, I am not sure about the rigorous definition for the direction of a gradient vector.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I am having trouble understanding what the sticking point is here, since you are demonstrating a reasonably advanced understanding of maths used in ML (elsewhere, as well as here). How much do you already know about vectors? Are you aware that a vector can be described by a direction and a maginitude? Your 1D example is a standard simplified version of that, applied to a 1D gradient. What is it, about vectors, or about gradients specifically, that is blocking you from understanding the same concept when handling gradients? $\endgroup$
    – Neil Slater
    Commented Nov 7, 2021 at 9:16
  • $\begingroup$ @NeilSlater I am exactly asking for a rigorous definition for the direction that we obtain from a gradient vector. Is it an angle wrt some axis? Is it another (unit) vector? Is it some other quantity? Is it a result of the vector addition of directions of wrt several axes?..... $\endgroup$
    – hanugm
    Commented Nov 7, 2021 at 9:37
  • $\begingroup$ @NeilSlater Please tell me in case of any ambiguity. $\endgroup$
    – hanugm
    Commented Nov 7, 2021 at 9:46
  • $\begingroup$ AFAIK, both the angles wrt axes, and unit normalised vector are valid and equivalent measures of a vector's direction, and apply to gradients (with no difference in this concept because the vector being described is a gradient). However, the request for formalism means I cannot really answer. A truly formal and robust definition may be more involved. $\endgroup$
    – Neil Slater
    Commented Nov 7, 2021 at 9:57
  • $\begingroup$ @NeilSlater Ha. I am updating the question for more clarity. Thanks for the suggestions. $\endgroup$
    – hanugm
    Commented Nov 7, 2021 at 9:58

1 Answer 1

Reset to default
1
$\begingroup$

If $u$ is a vector, the direction pointed by the vector is defined as $\dfrac{u}{\lVert {u}\rVert}$ where $\lVert \cdot \rVert$ is the 2 norm (euclidean norm).

Improve this answer
$\endgroup$
4
  • $\begingroup$ So, it may be a good idea to emphasize 2 things. 1. The gradient vector is not special. If a vector of $n$ real numbers, then it's a vector in $n$-d space and you cannot really visualize it if $n > 3$. 2. $\frac{u}{\|u\|}$ is the unit vector that has the same direction as $u$. Why do you compute the unit vector? Because you can have vectors of different magnitude that point in the same direction, and they are not the same vector because they have a different magnitude: e.g, in physics, you can have 2 vectors that represent 2 forces in the same direction but different magnitude. $\endgroup$
    – nbro
    Commented Nov 16, 2021 at 21:19
  • $\begingroup$ I think the definition for direction is becoming circular here. So, the direction pointed by a vector is the same as the direction pointed by the unit vector of that vector. But, what is meant by "direction pointed by a vector" mathematically? $\endgroup$
    – hanugm
    Commented Nov 16, 2021 at 22:30
  • $\begingroup$ @hanugm A vector is defined by a magnitude and direction. So, a vector cannot just be a line (i.e. without a direction) or a direction (i.e. without a magnitude). The definition may look circular indeed, because you use a vector to define the direction of another vector, but we can view this unit vector as representing the directions of infinitely many other vectors (if $u$ is a unit vector, then $c u$ all have the same direction, for all $c \in \mathbb{R}$), so this unit vector is not specific to the original vector you want to find the direction of. $\endgroup$
    – nbro
    Commented Jan 12, 2022 at 9:50
  • $\begingroup$ You could also view the direction as the angle that the vector makes with the axes. In that way, you might think that you don't need to define it with respect to a unit vector, but you would still need to define it with respect to a basis vector, so a vector. I'm not sure if there's any other "better" way to represent the direction of a vector. $\endgroup$
    – nbro
    Commented Jan 12, 2022 at 9:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .