I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help would be really helpful. Here are the questions:

  • Does the self-attention layer of Bert model have parameters? Do the embeddings of words change ONLY according to the actual embeddings of other words when the sentence is passed through the self-attention layer?

  • Are the parameters of the embedding layer of the model (the layer which transforms the sequence of indexes passed as input into a sequence of embeddings of size=size of the model) trainable or not?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.