In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):

$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$

where $w^s = (w_1^s,..., w_T^s)$ and $w^s_t$ is the word sampled from the model at time step $t$.

The derivation of the gradient of $L(\theta)$ concludes with Equation (7), where the gradient of $L(\theta)$ is approximated with a single sample $w^s \sim p_\theta$:

$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta} log \ p_{\theta}(w^s),$$

where $b$ is a reward baseline and $p_\theta(w^s)$ is the probability that sequence $w^s$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $L(\theta)$ w.r.t. the input of the softmax function $s_t$ (final layer):

$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$

I still understand the equation above.

And Equation (8):

$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$

where $1_{w^s_t}$ is $0$ everywhere, but $1$ at the $w^s_t$'th entry. How do you arrive at Equation (8)?

I'm happy to provide more information if necessary. In the paper "Sequence level training with recurrent neural networks", on page 7, they derive a similar result.


1 Answer 1


First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.
Loss is defined as:

$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$

If we use definition of expectation (for discrete case):

$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$
we get

$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$

probability $p_\theta(w_t)$ is defined as $softmax(s_t)$ = $S(s_t)$. If we calculate derivative of $L(\theta)$ with respect to $s_t$, we are basically calculating derivative of softmax $S(s_t)$ with respect to $s_t$.

It can be shown and i won't derive it here that derivative of softmax function is

$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$

where $1_{ij}$ here is defined as yours. If we plug this derivative into derivative of $L(\theta)$ and replace $S(s_t)$ with $p_\theta(w_t)$ (remember that probability distribution is defined as softmax here) we get:

$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$

we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:

$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$

now, if we ignore expectation we get:

$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$

I probably butchered notation is some places, but base idea should be clear

  • $\begingroup$ Okay nice I understand, thank you! Thanks for pointing out the mistake, it was a typo & I corrected it :-) $\endgroup$
    – rubencart
    Feb 21, 2019 at 14:03
  • $\begingroup$ By $p_\theta(w_t)$, you mean $p_\theta(w_t^s)$? $\endgroup$
    – nbro
    Feb 21, 2019 at 14:46
  • $\begingroup$ I'm still confused about this: $w^s = (w_1^s,..., w_T^s)$ is a sequence (dim $T$) of words $w^s_t$. I know & understand the $S(s)$ derivative, but here $w^s$ and $w^s_t$ do not have the same dimensions and I don't really see how you would take the softmax of $w^s$... $p_\theta(w^s_t)$ is indeed $softmax(s_t)$ I think but $p_\theta(w^s)$ should rather be a product of probabilities (one for each word in the sequence) right? Now if it were a sum or if we would drop in a logarithm the other probabilities could be omitted because we they are constant deriving w.r.t. $s_t$ maybe? $\endgroup$
    – rubencart
    Feb 21, 2019 at 15:19
  • $\begingroup$ I'm not entirely sure, I only glanced over the paper, I didn't actually read it. It is possible that I made some mistake especially in these middle step which they don't show in the paper. Maybe instead of $p_\theta(w^s)$ it would go $p_\theta(w_t^s)$ but they would differ in $t$, maybe you could still replace that with expectation over $w^s$ where each $w_t^s$ would be sampled $\endgroup$
    – Brale
    Feb 21, 2019 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .