-1
$\begingroup$

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: enter image description here

In the picture I highlight the rotary embeddings.

How can get the rotary embeddings and how can I interpret the output? What means 32x LLamaDecoderLayer and in its round brakets are four layer plus LlamaRotaryEmbeddings?

It's possible to get the embeddings as the first hidden-state hidden_state[0] and I want to know, which hidden-state represents the rotary embeddings. Am I right, that there are several rotary embeddings?

Thanks in forward.

Best regards.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The 32x LLamaDecoderLayer refers to a specific configuration of the LLamaDecoder model. It consists of 32 layers, each of which includes LlamaRotaryEmbeddings. The round brackets indicate that there are four layers in total, including the LlamaRotaryEmbeddings layer.

Regarding the hidden states, the first hidden state hidden_state[0] represents the output of the first layer in the model. Each subsequent hidden state hidden_state[i] represents the output of the i-th layer in the model. Therefore, hidden_state[0] does not specifically represent the rotary embeddings, but rather the output of the first layer in general. There can be multiple rotary embeddings depending on the specific configuration and implementation of the model.

https://blog.eleuther.ai/rotary-embeddings/

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. If I take the last hidden state, are that the rotary embeddings? $\endgroup$
    – Christian01
    Commented Sep 29, 2023 at 14:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .