2
$\begingroup$

I am reading the BERT paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

As I look at the attention mechanism, I don't understand why in the BERT encoder we have an intermediate layer between the attention and neural network layers with a bigger output ($4*H$, where $H$ is the hidden size). Perhaps it is the layer normalization, but, by looking at the code, I'm not certain.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.