Recently I was reading a paper based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training with BLEU and SIMILE.

What I'm unable to understand is what is meant by the expected BLEU cost when training with BLEU and SIMILE? Are there any separate cost functions defined for these scores?

I'm attaching the image of the graph.

enter image description here


It looks like the method they use for training takes a set of candidate hypotheses $\mathcal{U}(x)$, along with associated probabilities, and then minimizes the expected value of the cost function over that distribution. Section 3 has the loss function being minimized:

$ \mathcal{L}_{Risk} = \sum\limits_{u \in \mathcal{U}(x)} cost(t, u) \frac{p(u|x)}{\sum_{u' \in \mathcal{U}(x)} p(u'|x)} $.

One of the cost functions used is $1 - \texttt{BLEU}(t, h)$, where $t$ is the target and $h$ is the generated hypothesis. I'm not sure where $p(u|x)$ is coming from, but $1 - \mathcal{L}_{Risk}$ for the BLEU cost function is probably what they're refering to when they mention Expected BLEU.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.