In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward of a generated sequence of words (Equation (3)):

$$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)],$$

where $w^s = (w_1^s,..., w_T^s)$ and $w^s_t$ is the word sampled from the model at time step $t$.

The derivation of the gradient of $L(\theta)$ concludes with Equation (7), where the gradient of $L(\theta)$ is approximated with a single sample $w^s \sim p_\theta$:

$$\nabla_{\theta}L(\theta) \approx -(r(w^s) - b) \ \triangledown_{\theta} log \ p_{\theta}(w^s),$$

where $b$ is a reward baseline and $p_\theta(w^s)$ is the probability that sequence $w^s$ is sampled from the model. Up until here I understand what's going on. However, then they proceed with defining the partial derivative of $L(\theta)$ w.r.t. the input of the softmax function $s_t$ (final layer):

$$\nabla_{\theta}L(\theta) = \sum^T_{t=1} \frac{\partial L(\theta)}{\partial s_t} \frac{\partial s_t}{\partial \theta}$$

I still understand the equation above.

And Equation (8):

$$\frac{\partial L(\theta)}{\partial s_t} \approx (r(w^s) - b) (p_\theta(w_t| h_t) - 1_{w^s_t}),$$

where $1_{w^s_t}$ is $0$ everywhere, but $1$ at the $w^s_t$'th entry. How do you arrive at Equation (8)?

I'm happy to provide more information if necessary. In the paper "Sequence level training with recurrent neural networks", on page 7, they derive a similar result.


First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.
Loss is defined as:

$L(\theta) = - \mathbb{E}_{w^s \sim p_{\theta}}[r(w^s)]$

If we use definition of expectation (for discrete case):

$\mathbb{E}[X] = \sum\limits_{i} p_i(x_i)x_i$
we get

$L(\theta) = -\sum\limits_{w^s} p_\theta(w^s)r(w^s)$

probability $p_\theta(w_t)$ is defined as $softmax(s_t)$ = $S(s_t)$. If we calculate derivative of $L(\theta)$ with respect to $s_t$, we are basically calculating derivative of softmax $S(s_t)$ with respect to $s_t$.

It can be shown and i won't derive it here that derivative of softmax function is

$\frac{\partial S(s_i)}{\partial s_j} = S(s_i)(1_{ij} - S(s_j))$

where $1_{ij}$ here is defined as yours. If we plug this derivative into derivative of $L(\theta)$ and replace $S(s_t)$ with $p_\theta(w_t)$ (remember that probability distribution is defined as softmax here) we get:

$\frac{\partial L(\theta)}{\partial s_t} = -\sum\limits_{w^s} p_\theta(w^s)(1_{w^s_t} - p_\theta(w_t))r(w^s)$

we extract the - from the equation above, use definition of expectation again and subtract baseline from reward and we have:

$\frac{\partial L(\theta)}{\partial s_t} = \mathbb{E}_{w^s \sim p_{\theta}}[(p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)]$

now, if we ignore expectation we get:

$\frac{\partial L(\theta)}{\partial s_t} \approx (p_\theta(w_t) - 1_{w^s_t})(r(w^s) - b)$

I probably butchered notation is some places, but base idea should be clear

  • $\begingroup$ Okay nice I understand, thank you! Thanks for pointing out the mistake, it was a typo & I corrected it :-) $\endgroup$
    – rubencart
    Feb 21 '19 at 14:03
  • $\begingroup$ By $p_\theta(w_t)$, you mean $p_\theta(w_t^s)$? $\endgroup$
    – nbro
    Feb 21 '19 at 14:46
  • $\begingroup$ I'm still confused about this: $w^s = (w_1^s,..., w_T^s)$ is a sequence (dim $T$) of words $w^s_t$. I know & understand the $S(s)$ derivative, but here $w^s$ and $w^s_t$ do not have the same dimensions and I don't really see how you would take the softmax of $w^s$... $p_\theta(w^s_t)$ is indeed $softmax(s_t)$ I think but $p_\theta(w^s)$ should rather be a product of probabilities (one for each word in the sequence) right? Now if it were a sum or if we would drop in a logarithm the other probabilities could be omitted because we they are constant deriving w.r.t. $s_t$ maybe? $\endgroup$
    – rubencart
    Feb 21 '19 at 15:19
  • $\begingroup$ I'm not entirely sure, I only glanced over the paper, I didn't actually read it. It is possible that I made some mistake especially in these middle step which they don't show in the paper. Maybe instead of $p_\theta(w^s)$ it would go $p_\theta(w_t^s)$ but they would differ in $t$, maybe you could still replace that with expectation over $w^s$ where each $w_t^s$ would be sampled $\endgroup$
    – Brale
    Feb 21 '19 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.