How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?

Question

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):

$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$

where $w^s = (w_1^s,..., w_T^s)$ and $w^s_t$ is the word sampled from the model at time step $t$.

The derivation of the gradient of $L(\theta)$ concludes with Equation (7), where the gradient of $L(\theta)$ is approximated with a single sample $w^s \sim p_\theta$:

$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta} log \ p_{\theta}(w^s),$$

where $b$ is a reward baseline and $p_\theta(w^s)$ is the probability that sequence $w^s$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $L(\theta)$ w.r.t. the input of the softmax function $s_t$ (final layer):

$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$

I still understand the equation above.

And Equation (8):

$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$

where $1_{w^s_t}$ is $0$ everywhere, but $1$ at the $w^s_t$'th entry. How do you arrive at Equation (8)?

I'm happy to provide more information if necessary. In the paper "Sequence level training with recurrent neural networks", on page 7, they derive a similar result.

Answer 1

First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.
Loss is defined as:

$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$

If we use definition of expectation (for discrete case):

$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$
we get

$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$

probability $p_\theta(w_t)$ is defined as $softmax(s_t)$ = $S(s_t)$. If we calculate derivative of $L(\theta)$ with respect to $s_t$, we are basically calculating derivative of softmax $S(s_t)$ with respect to $s_t$.

It can be shown and i won't derive it here that derivative of softmax function is

$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$

where $1_{ij}$ here is defined as yours. If we plug this derivative into derivative of $L(\theta)$ and replace $S(s_t)$ with $p_\theta(w_t)$ (remember that probability distribution is defined as softmax here) we get:

$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$

we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:

$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$

now, if we ignore expectation we get:

$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$

I probably butchered notation is some places, but base idea should be clear