enter image description here

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.

ELMo paper - https://arxiv.org/pdf/1802.05365.pdf

BERT paper - https://arxiv.org/pdf/1810.04805.pdf

  • $\begingroup$ Can you please contextualise? What is ELMo? Link us to a paper/article which describes ELMo? How is the image related to the picture? You may also want to ask the question in the body of your post. $\endgroup$ – nbro Feb 17 at 11:26
  • $\begingroup$ @nbro search for BERT and ELMo NLP papers. $\endgroup$ – Skinish Feb 18 at 10:15
  • $\begingroup$ You should provide such info in your question, given that this are not very well known models, etc. $\endgroup$ – nbro Feb 18 at 11:30
  • $\begingroup$ @nbro in this case I have to disagree as they are both state of the art releases from last year. I am going to add the references nevertheless. $\endgroup$ – Skinish Feb 18 at 13:35
  • $\begingroup$ Not all people in the community have been able to read all papers from last year. $\endgroup$ – nbro Feb 18 at 13:38

These embeddings are nothing more than token embeddings.

You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.

  • 1
    $\begingroup$ Q: they train embeddings for the first sentence, second sentence etc? A: yes. $\endgroup$ – soloice Feb 19 at 11:03
  • 1
    $\begingroup$ Q:They must be limited to a max length right? A: position embeddings are limited to a max length (I forget the exact value, but you can find it in the paper), while there are only 2 sentence embeddings (E_A and E_B). $\endgroup$ – soloice Feb 19 at 11:05
  • 1
    $\begingroup$ Q: what is the dimension of these embeddings used? A: The same as token embedding. $\endgroup$ – soloice Feb 19 at 11:05
  • 1
    $\begingroup$ Q: how do they combine all the 3 type of embeddings? A: Add them up. $\endgroup$ – soloice Feb 19 at 11:06
  • 1
    $\begingroup$ Yes, you'll add the same learned segment embedding to all token embeddings. But it isn't dummy. It tells the model "you should treat all input tokens identically/you can encode the inputs as a whole instead of extracting features from the first and second half separately." If you don't do this, you'll create a train/test discrepancy, because the model hasn't seen any example without segment embedding. $\endgroup$ – soloice Feb 21 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.