# What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to output fully connected layer for the encoder just after adding the residual connection.

Why is there an intermediate layer within an encoder block?

For example,

encoder.layer.11.attention.self.query.weight
encoder.layer.11.attention.self.query.bias
encoder.layer.11.attention.self.key.weight
encoder.layer.11.attention.self.key.bias
encoder.layer.11.attention.self.value.weight
encoder.layer.11.attention.self.value.bias
encoder.layer.11.attention.output.dense.weight
encoder.layer.11.attention.output.dense.bias
encoder.layer.11.attention.output.LayerNorm.weight
encoder.layer.11.attention.output.LayerNorm.bias
encoder.layer.11.intermediate.dense.weight
encoder.layer.11.intermediate.dense.bias

encoder.layer.11.output.dense.weight
encoder.layer.11.output.dense.bias
encoder.layer.11.output.LayerNorm.weight
encoder.layer.11.output.LayerNorm.bias

I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers

• Welcome to SE:AI! I did a slight edit to draw attention to the question posed.
– DukeZhou
Oct 26 '21 at 0:22
• Hello. Can you please provide the link to the PyTorch model that has an intermediate layer that you're referring to?
– nbro
Oct 26 '21 at 11:41
• Hello thank you for your response. in fact each pre-trained model on hugging face has the same architecture. you can look at the following for example 'bert-base-uncased' at huggingface.co/bert-base-uncased (three fully connected layers. one in attention one intermediate and one output. for all encoder blocks) Oct 26 '21 at 12:15
• I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers. Oct 26 '21 at 12:25

Transformer architecture, in addition to the self-attention layer, that aggregates information from the whole sequence and transforms each token due to the attention scores from the queries and values has a feedforward layer, which is mostly a 2-layer MLP, that processes each token separately: $$y = W_2 \sigma(W_1 x + b_1) + b_2$$
Where $$W_1, W_2$$ are the weights, and $$b_1, b_2$$ - biases, $$\sigma$$ - is the nonlinearity ReLU, GeLU, e.t.c.
I suspect, that intermediate here corresponds to $$W_1, b_1$$ and the output is about $$W_2, b_2$$.