# Where does the expectation term in the derivative of the soft-max policy come from?

At slide 17 of the David Silver's series, the soft-max policy is defined as follows

$$\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}$$

that is, the probability of an action $$a$$ (in state $$s$$) is proportional to the "exponentiated weight".

The score function is then defined as follows

$$\nabla_\theta \log \pi_\theta (s, a) = \phi(s, a) - \mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]$$

Where does the expectation term $$\mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]$$ come from?

• I'm not 100% sure what you are asking but I'll try to answer it anyway. The expectation term comes from the derivative of the softmax function. This video from Geoffrey Hinton should help: youtube.com/watch?v=mlaLLQofmR8 – Jaden Travnik Jan 17 '19 at 16:15
• The confusion here is that, when we take log of pi, then it will log(sum x_i) in dem, where x_i is that expression. Now my doubt is here this term log(sum_i x_i) is replace by sum_i (log x_i), how? Is this inequality valid? – Ankish Bansal Jan 17 '19 at 17:58
• see this answer over on the math.stackexchange.com site: math.stackexchange.com/a/2340848/414001 – Dennis Soemers Feb 15 '19 at 19:06