# Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than the output/softmax vector? Why is the shape of the softmax vector's derivative (the Jacobian) different than the shape of the derivative of the other activation functions, such as the ReLU and sigmoid?

• This blog post could also be useful.
– nbro
Jan 2 '21 at 20:43

If you have a function $$f:\mathbb{R}\rightarrow\mathbb{R}$$ the derivative is a function on its own and you have $$f':\mathbb{R}\rightarrow\mathbb{R}.$$ If you increase the dimension of the input space have $$f:\mathbb{R}^n\rightarrow\mathbb{R}.$$ The "derivative" in this case is called gradient and it is a vector collecting the $$n$$ partial derivatives of $$f$$. The input space of the gradient function is $$\mathbb{R}^n$$ (the same as for $$f$$), but the output is the collection of the $$n$$ derivatives, so the output space is also $$\mathbb{R}^n$$. In other words $$\nabla f:\mathbb{R}^n\rightarrow\mathbb{R}^n,$$ which makes sense as for each point $$x$$ of the input space you get a vector ($$\nabla f(x)$$) as output.
So far so good, but what happens if you consider a function that takes a vector as input and spits out a vector as output, i.e. $$f:\mathbb{R}^n\rightarrow\mathbb{R}^m?$$ How to compute the equivalent of the derivative? (This is the softmax case, where you have a vector as input and a vector as output.)
You can reduce this case to the previous case considering $$f=(f_1, \dots, f_m)$$, where $$f_i:\mathbb{R}^n\rightarrow\mathbb{R}.$$ Now for each $$f_i$$ you can compute the gradient $$\nabla f_i$$ and end up with $$m$$ gradients. When you evaluate them at a point $$x\in\mathbb{R}^n$$ you get $$m$$ $$n-$$dimensional vectors. These vector can be collected in a matrix, which is the Jacobian; formally $$Jf:\mathbb{R}^n\rightarrow\mathbb{R}^{m\times n}.$$ Finally, to answer your question, you get a Jacobian "instead" of a gradient (they all represent the same concept) because the output of the softmax is not a single number but a vector.
By the way the sigmoid and relu are functions with one dimensional input and output, so they don't really have a gradient but a derivative. The trick is that people write $$\sigma(W)$$, where $$W$$ is vector or a matrix, but they mean that $$\sigma$$ is applied component-wise, as a function from $$\mathbb{R}$$ to $$\mathbb{R}$$ (I know, it's confusing).