I am reading the BERT paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output ($4*H$, where $H$ is the hidden size). Perhaps it is the layer normalization, but, by looking at the code, I'm not certain.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.