In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):
$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$
where $w^s = (w_1^s,..., w_T^s)$ and $w^s_t$ is the word sampled from the model at time step $t$.
The derivation of the gradient of $L(\theta)$ concludes with Equation (7), where the gradient of $L(\theta)$ is approximated with a single sample $w^s \sim p_\theta$:
$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta} log \ p_{\theta}(w^s),$$
where $b$ is a reward baseline and $p_\theta(w^s)$ is the probability that sequence $w^s$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $L(\theta)$ w.r.t. the input of the softmax function $s_t$ (final layer):
$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$
I still understand the equation above.
And Equation (8):
$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$
where $1_{w^s_t}$ is $0$ everywhere, but $1$ at the $w^s_t$'th entry. How do you arrive at Equation (8)?
I'm happy to provide more information if necessary. In the paper "Sequence level training with recurrent neural networks", on page 7, they derive a similar result.