I have trouble understanding how to implement derivative of softmax function. Here is what I tried:

def Softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

def d_Softmax(X):
    return (np.diagflat(s) - np.dot(s, s.T))

I am not sure if it works as it should. The normal softmax function, when taking $m \times n$ matrix as input, outputs a matrix of the same shape. The derivative, however, seems to output a matrix of shape $mn \times mn$.

I am trying to implement backpropagation of a simple 3-layer neural network on my own, but no other matrix has the shape aligned with the derivative that the softmax returns, so I don't know what it should be multiplied with.

  • $\begingroup$ Programming issues are off-topic here. Please, read ai.stackexchange.com/help/on-topic for more details and to understand which questions are on-topic here. If you have a programming issue, the best place to ask your question is Stack Overflow or, alternatively, Data Science SE. $\endgroup$ – nbro Jun 2 at 22:21