# Tag Info

1 vote

### Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data

In order for a neural network to generalize well, it has to be trained on tons and tons of data, otherwise if the model is shown very few samples of the training data, it will be biased towards these ...
• 425
Accepted

### What should be Relationship between embedding dimension and context length?

Embedding Dimension (d_model): One word/token will be represented as "d_model" different numebrs. Larger the d_model,higher thwe capacityof storing the true meaning of the word It also ...

### xLSTM parallel computation - mismatch in dimensions

$$Q \in R^{T \times d}, K \in R^{T \times d} \\ Q \cdot K^T \in R^{T \times d} \times R^{d \times T}\\ Q \cdot K^T \in R^{T \times T}$$ Which is exactly the dimension of $D$
• 2,133

### What if the sum of word embedding and positional embedding becomes same for different words?

This would be incredibly difficult just due to the size of these embedding vectors and the fact that we are performing 2D rotations (many in parallel). The larger this embedding space the more the ...
• 11
1 vote

### Last linear layer of the decoder of a transformer

Well well, this text in the link posted in one of the answers above (https://www.tensorflow.org/text/tutorials/transformer) gives the answer to the question. Note: The model is optimized for ...
• 111
Accepted

### Understanding different methods of covariance parametrization

Parametrizing something is literally what the term is saying: "how do you put parameters so that you can end up having it via such parameters" For example, you can parametrize the covariance ...
• 2,133

### What if the sum of word embedding and positional embedding becomes same for different words?

The positional embedding values in transformers are fixed from the start, they are calculated only once before the start of training (or stored/reused from years ago in memory). So there is only very ...
• 145
Accepted

### What does "position-wise" fully connected mean?

Position-Wise basically means that the architecture takes position of a given value in the vector into considiration, and changing position of a given value in the input will impact the output.
Accepted

### Is there a relationship between tokens and parameters in LLMs?

Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt) Consider that the context window that you see in the ...
• 2,133