# How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $$\theta$$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):

$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$

where $$w^s = (w_1^s,..., w_T^s)$$ and $$w^s_t$$ is the word sampled from the model at time step $$t$$.

The derivation of the gradient of $$L(\theta)$$ concludes with Equation (7), where the gradient of $$L(\theta)$$ is approximated with a single sample $$w^s \sim p_\theta$$:

$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta} log \ p_{\theta}(w^s),$$

where $$b$$ is a reward baseline and $$p_\theta(w^s)$$ is the probability that sequence $$w^s$$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $$L(\theta)$$ w.r.t. the input of the softmax function $$s_t$$ (final layer):

$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$

I still understand the equation above.

And Equation (8):

$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$

where $$1_{w^s_t}$$ is $$0$$ everywhere, but $$1$$ at the $$w^s_t$$'th entry. How do you arrive at Equation (8)?

I'm happy to provide more information if necessary. In the paper "Sequence level training with recurrent neural networks", on page 7, they derive a similar result.

First of all you made a mistake, equation 8 in the paper is defined with $$\frac{\partial L(\theta)}{\partial s_t}$$ not $$\frac{\partial L(\theta)}{\partial\theta}$$.
Loss is defined as:

$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$$

If we use definition of expectation (for discrete case):

$$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$$
we get

$$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$$

probability $$p_\theta(w_t)$$ is defined as $$softmax(s_t)$$ = $$S(s_t)$$. If we calculate derivative of $$L(\theta)$$ with respect to $$s_t$$, we are basically calculating derivative of softmax $$S(s_t)$$ with respect to $$s_t$$.

It can be shown and i won't derive it here that derivative of softmax function is

$$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$$

where $$1_{ij}$$ here is defined as yours. If we plug this derivative into derivative of $$L(\theta)$$ and replace $$S(s_t)$$ with $$p_\theta(w_t)$$ (remember that probability distribution is defined as softmax here) we get:

$$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$$

we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:

$$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$$

now, if we ignore expectation we get:

$$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$$

I probably butchered notation is some places, but base idea should be clear

• Okay nice I understand, thank you! Thanks for pointing out the mistake, it was a typo & I corrected it :-) Feb 21 '19 at 14:03
• By $p_\theta(w_t)$, you mean $p_\theta(w_t^s)$?
– nbro
Feb 21 '19 at 14:46
• I'm still confused about this: $w^s = (w_1^s,..., w_T^s)$ is a sequence (dim $T$) of words $w^s_t$. I know & understand the $S(s)$ derivative, but here $w^s$ and $w^s_t$ do not have the same dimensions and I don't really see how you would take the softmax of $w^s$... $p_\theta(w^s_t)$ is indeed $softmax(s_t)$ I think but $p_\theta(w^s)$ should rather be a product of probabilities (one for each word in the sequence) right? Now if it were a sum or if we would drop in a logarithm the other probabilities could be omitted because we they are constant deriving w.r.t. $s_t$ maybe? Feb 21 '19 at 15:19
• I'm not entirely sure, I only glanced over the paper, I didn't actually read it. It is possible that I made some mistake especially in these middle step which they don't show in the paper. Maybe instead of $p_\theta(w^s)$ it would go $p_\theta(w_t^s)$ but they would differ in $t$, maybe you could still replace that with expectation over $w^s$ where each $w_t^s$ would be sampled Feb 21 '19 at 16:07