6
$\begingroup$

The paper Attention Is All You Need describes the transformer architecture that has an encoder and a decoder.

However, I wasn't clear on what the cost function to minimize is for such an architecture.

Consider a translation task, for example, where give an English sentence $x_{english} = [x_0, x_1, x_2, \dots, x_m]$, the transformer decodes the sentence into a French sentence $x_{french}' = [x_0', x_1', \dots, x_n']$. Let's say the true label is $y_{french} = [y_0, y_1, \dots, y_p]$.

What is the object function of the transformer? Is it the MSE between $x_{french}'$ and $y_{french}$? And does it have any weight regularization terms?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
7
$\begingroup$

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\text{vocab}\|$ (right before taking the argmax to find the token to output), and the $\|\text{sentence length}\|$-length vector of token IDs as the true label.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Note that they reshape the out_logits variable there, in the line that you're linking us to, i.e. tf.reshape(out_logits, [-1, VOCAB_SIZE]), so the cross-entropy seems to be computed between two vectors and not a matrix and a vector (but I didn't really executed that code and tried to output the shape of those tensors/variables). It may be a good idea to do it, just to confirm the exact shape. $\endgroup$
    – nbro
    Dec 10, 2020 at 10:21
  • 1
    $\begingroup$ @nbro I just ran the code, and can confirm that is the case. out_logits originally has the shape (3, 7, 1, 1, 10), which is to (BATCH_SIZE, SENTENCE_LENGTH, ??, ??, VOCAB_SIZE). The ?? are bookkeeping dimensions because the tensor2tensor model expects a 5-D tensor, but are then immediately stripped out afterwards. Then, tf.reshape(out_logits, [-1, VOCAB_SIZE]) resizes out_logits to (21, 10), which is (BATCH_SIZE * SENTENCE_LENGTH, VOCAB_SIZE). Btw, for tf.reshape, seems like adding a -1 as a dimension makes it a flexible sized dimension, so it results in a 2D matrix rather than a 1D vector. $\endgroup$
    – user3667125
    Dec 13, 2020 at 0:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .