# How do I derive the gradient with respect to the parameters of the softmax policy?

The gradient of the softmax eligibility trace is given by the following:

\begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align}

How is this equation derived?

The following relation is true:

\begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} \tag{1}\label{1} \end{align}

Thus, the following relation must also be true: \begin{align} \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} &=\phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align}

Mathematically, why would this be the case? Probably, you just need to answer my question above because \ref{1} is true and it's just the rule to differentiate a logarithm.

Softmax policy $$\pi_\theta(s,a)$$ is defined as $$\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$$, where the summation is over the action space.
Taking log, this becomes $$\log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\ = \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }})$$
Taking derivative wrt $$\theta$$, this becomes $$\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \nabla_\theta log({\Sigma e^{\phi(s,a) ^T \theta }})$$
We can rewrite $$\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$$ as follows. $$\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }}) = \frac{\nabla_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}} = \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}} = \Sigma \phi(s,a) \pi_\theta(s,a)$$
The final equation then becomes $$\nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi_\theta(s,a)$$
• I'd recommend writing each of the $e^{\phi(s, a) \theta}$ things as $\exp(\phi(s, a)^{\top} \theta)$. Not having to write it in the smaller superscript makes it a bit more readable, especially for the in-text equations. And I like making the transpose a bit more explicitly visible too personally :) That last part is assuming we're dealing with feature and weight vectors of course, not just scalars May 21, 2020 at 19:36