The gradient of the softmax eligibility trace is given by the following:

\begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align}

How is this equation derived?

The following relation is true:

\begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} \tag{1}\label{1} \end{align}

Thus, the following relation must also be true: \begin{align} \frac{\nabla_{\theta} \pi_{\theta}(a|s)}{\pi_{\theta}(a|s)} &=\phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align}

Mathematically, why would this be the case? Probably, you just need to answer my question above because \ref{1} is true and it's just the rule to differentiate a logarithm.


Softmax policy $\pi_\theta(s,a)$ is defined as $\frac{\exp{(\phi(s,a)^T \theta})}{\Sigma \exp{(\phi(s,a) ^T \theta) }}$, where the summation is over the action space.
Taking log, this becomes $$ \log \pi_\theta(s,a) = log(e^{\phi(s,a) ^T \theta}) - log({\Sigma e^{\phi(s,a) ^T \theta }}) \\ = \phi(s,a) ^T \theta - log({\Sigma e^{\phi(s,a)^T \theta }}) $$

Taking derivative wrt $\theta$, this becomes $$ \nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \nabla_\theta log({\Sigma e^{\phi(s,a) ^T \theta }}) $$

We can rewrite $\nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }})$ as follows. $$ \nabla_\theta log({\Sigma e^{\phi(s,a)^T \theta }}) = \frac{\nabla_\theta \Sigma e^{\phi(s,a)^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}} = \frac{\Sigma \phi(s,a) e^{\phi(s,a) ^T \theta}}{\Sigma e^{\phi(s,a) ^T \theta}} = \Sigma \phi(s,a) \pi_\theta(s,a) $$

The final equation then becomes $$ \nabla_\theta \log \pi_\theta(s,a) = \phi(s,a) - \Sigma \phi(s,a) \pi_\theta(s,a) $$

| improve this answer | |
  • 1
    $\begingroup$ I'd recommend writing each of the $e^{\phi(s, a) \theta}$ things as $\exp(\phi(s, a)^{\top} \theta)$. Not having to write it in the smaller superscript makes it a bit more readable, especially for the in-text equations. And I like making the transpose a bit more explicitly visible too personally :) That last part is assuming we're dealing with feature and weight vectors of course, not just scalars $\endgroup$ – Dennis Soemers May 21 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.