enter image description here

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.

ELMo paper - https://arxiv.org/pdf/1802.05365.pdf

BERT paper - https://arxiv.org/pdf/1810.04805.pdf

  • $\begingroup$ Can you please contextualise? What is ELMo? Link us to a paper/article which describes ELMo? How is the image related to the picture? You may also want to ask the question in the body of your post. $\endgroup$ – nbro Feb 17 '19 at 11:26
  • $\begingroup$ @nbro search for BERT and ELMo NLP papers. $\endgroup$ – Skinish Feb 18 '19 at 10:15
  • $\begingroup$ You should provide such info in your question, given that this are not very well known models, etc. $\endgroup$ – nbro Feb 18 '19 at 11:30
  • $\begingroup$ @nbro in this case I have to disagree as they are both state of the art releases from last year. I am going to add the references nevertheless. $\endgroup$ – Skinish Feb 18 '19 at 13:35
  • $\begingroup$ Not all people in the community have been able to read all papers from last year. $\endgroup$ – nbro Feb 18 '19 at 13:38

These embeddings are nothing more than token embeddings.

You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.

| improve this answer | |
  • 2
    $\begingroup$ Q: they train embeddings for the first sentence, second sentence etc? A: yes. $\endgroup$ – soloice Feb 19 '19 at 11:03
  • 2
    $\begingroup$ Q:They must be limited to a max length right? A: position embeddings are limited to a max length (I forget the exact value, but you can find it in the paper), while there are only 2 sentence embeddings (E_A and E_B). $\endgroup$ – soloice Feb 19 '19 at 11:05
  • 2
    $\begingroup$ Q: what is the dimension of these embeddings used? A: The same as token embedding. $\endgroup$ – soloice Feb 19 '19 at 11:05
  • 2
    $\begingroup$ Q: how do they combine all the 3 type of embeddings? A: Add them up. $\endgroup$ – soloice Feb 19 '19 at 11:06
  • 1
    $\begingroup$ Yes, you'll add the same learned segment embedding to all token embeddings. But it isn't dummy. It tells the model "you should treat all input tokens identically/you can encode the inputs as a whole instead of extracting features from the first and second half separately." If you don't do this, you'll create a train/test discrepancy, because the model hasn't seen any example without segment embedding. $\endgroup$ – soloice Feb 21 '19 at 6:12

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:

  • First, a [SEP] token is put between them
  • Second, a learned embedding $E_A$ is concatenated to every token of the first sentence, and another learned vector $E_B$ to every token of the second one

That is, there are just two possible "segment embeddings" (for the first sentence and for the second one).

Positional embeddings are learned vectors for every possible position between 0 and 512-1. Transformers don't have a sequential nature as recurrent neural networks, so some information about the order of the input is needed; if you disregard this, your output will be permutation-invariant.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.