# Is this the correct gradient for log of softmax? [duplicate]

I am currently implementing the very basic version (REINFORCE) of the Monte Carlo policy gradient algorithm. I was wondering if this is the correct gradient for the log of softmax.

\begin{align} \nabla_{\theta} \log \pi_{\theta}(s, a) &= \varphi(s, a)-\mathbb{E}\left[\varphi(s, a)_{\forall a \in A}\right] \\ &= \left(\varphi(s)^T \cdot \theta_{a}\right)-\sum_{\forall a \in A}\left(\varphi(s)^T \cdot \theta_{a}\right) \end{align}

where $$\varphi(s)$$ is the feature vector at state $$s$$.

I am not sure if my interpretation of the equation is correct. I ask because, in my implementation, my weights ($$\theta$$) blow up after a few iterations, and I have a feeling the problem is in this line.