At slide 17 of the David Silver's series, the soft-max policy is defined as follows

$$ \pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta} $$

that is, the probability of an action $a$ (in state $s$) is proportional to the "exponentiated weight".

The score function is then defined as follows

$$ \nabla_\theta \log \pi_\theta (s, a) = \phi(s, a) - \mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)] $$

Where does the expectation term $\mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]$ come from?

  • $\begingroup$ I'm not 100% sure what you are asking but I'll try to answer it anyway. The expectation term comes from the derivative of the softmax function. This video from Geoffrey Hinton should help: youtube.com/watch?v=mlaLLQofmR8 $\endgroup$ Jan 17 '19 at 16:15
  • $\begingroup$ The confusion here is that, when we take log of pi, then it will log(sum x_i) in dem, where x_i is that expression. Now my doubt is here this term log(sum_i x_i) is replace by sum_i (log x_i), how? Is this inequality valid? $\endgroup$ Jan 17 '19 at 17:58
  • 1
    $\begingroup$ see this answer over on the math.stackexchange.com site: math.stackexchange.com/a/2340848/414001 $\endgroup$
    – Dennis Soemers
    Feb 15 '19 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.