1
$\begingroup$

I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help would be really helpful. Here are the questions:

  • Does the self-attention layer of Bert model have parameters? Do the embeddings of words change ONLY according to the actual embeddings of other words when the sentence is passed through the self-attention layer?

  • Are the parameters of the embedding layer of the model (the layer which transforms the sequence of indexes passed as input into a sequence of embeddings of size=size of the model) trainable or not?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.