I have had a look at LLamas model card, specifically the 7B parameter version: https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md
which I assume is an encoder only transformer similar to this:
But then I did some math.If the dimension of every Dense layer, including the one connecting to the Attention layer is 4096, the context length is 2048, the number of attention heads is 32 and the embedding size is 786, then the output size of the attention layer is 32 * 786 * 2048 and as such the number of weights to connect it to the dense layer is 32 * 766 * 2048 * 4096, which is 205B parameters, which is obviously far more than 7B. So how is this accomplished? How big is the ouput of the attention layer and how is it connected to the following Dense layers?