4
$\begingroup$

I read the answers for that question What kind of word embedding is used in the original transformer?. It says that transforms like bert start the first word embedding layer with random values.

Initializing the first word embedding layer in transformers with random values works fine but Wouldn't initializing transformers with pre-trained word embedding speed up the training of transformers?

Isn't starting with pre-trained word embedding(vectors that have semantic meaning) is better than starting from scratch?

I am not talking about the performance. I am talking about the speed of the training.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I'd expect your approach to converge faster indeed.

Not having to start from scratch regarding your embedding vectors means that there are fewer parameters to learn. Assuming that the token embeddings are indeed useful for the model, the model should converge faster. However, note that a trainable token embedding might eventually exceed the model with a pre-trained token embedding with regard to performance.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What about a fine-tuning approach, whereby the original pre-trained token/word embeddings can be finetuned to the task $\endgroup$
    – information_interchange
    Commented Jul 8, 2023 at 18:01
  • 1
    $\begingroup$ You migjt then wany to start with frozen embeddings, and slowly unfreeze them. Otherwise you might get some catastrophic forgetting $\endgroup$
    – Robin van Hoorn
    Commented Jul 9, 2023 at 8:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .