I have trouble understanding how to implement derivative of softmax function. Here is what I tried:
def Softmax(x): e_x = np.exp(x - np.max(x)) return e_x / e_x.sum() def d_Softmax(X): x=Softmax(X) s=x.reshape(-1,1) return (np.diagflat(s) - np.dot(s, s.T))
I am not sure if it works as it should. The normal softmax function, when taking $m \times n$ matrix as input, outputs a matrix of the same shape. The derivative, however, seems to output a matrix of shape $mn \times mn$.
I am trying to implement backpropagation of a simple 3-layer neural network on my own, but no other matrix has the shape aligned with the derivative that the softmax returns, so I don't know what it should be multiplied with.