How is the Jacobian a generalisation of the gradient?

Question

I came across these slides Natural Language Processing with Deep Learning CS224N/Ling284, in the context of natural language processing, which talk about the Jacobian as a generalization of the gradient.

I know there is a lot of topic regarding this on the internet, and trust me, I've googled it. But things are getting more and more confused for me.

In simple words, how is the Jacobian a generalization of the gradient? How can it be used in gradient descent?

Answer 1

In short, the Jacobian matrix is a generalization of the gradient for vector-valued functions.

Recall that the gradient is a vector of partial derivatives of a multi-variable function. So, consider a multi-variable function of the form $f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}$. The output of this function is $f(x_1, x_2, \dots, x_N) = y$, where $x_i \in \mathcal{X}_i$, for $i=1, \dots, N$ and $y \in \mathcal{Y}$. And the gradient is $\nabla f = \left[ \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_N} \right] \in \mathbb{R}$.

A vector-valued function is a function whose output is a vector, i.e. a function of the form $f: \mathcal{X} \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M$ (I am not sure if this notation is rigorous enough!), so the output of this function is a vector $f(x) = [y_1, y_2, \dots, y_M]$, where $x \in \mathcal{X}$ and $y_i \in \mathcal{Y}_i$, for $i = 1, \dots, M$. You can also view a vector-valued function $f$ as a vector of scalar-valued functions $[f_1, f_2, \dots, f_M]$, where $f_i: \mathcal{X} \rightarrow \mathcal{Y}_i$, for all $i$.

You can also have multi-variable vector-valued functions, i.e. functions of the form

$$f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M.$$

The Jacobian matrix is an $N \times M$ matrix with one partial derivative for each combination of inputs and outputs (i.e. $f_i$).

If you want to optimize a multi-variable vector-valued function, you can make use of the Jacobian, in a similar way that you make use of the gradient in the case of multi-variable functions, but, although I've seen it in the past, I can't provide now a concrete example of an application of the Jacobian (but the linked slides probably do that).