Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

New answers tagged

1 vote

Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data

In order for a neural network to generalize well, it has to be trained on tons and tons of data, otherwise if the model is shown very few samples of the training data, it will be biased towards these ...
Leo's user avatar
  • 425
0 votes
Accepted

What should be Relationship between embedding dimension and context length?

Embedding Dimension (d_model): One word/token will be represented as "d_model" different numebrs. Larger the d_model,higher thwe capacityof storing the true meaning of the word It also ...
Kulin Patel's user avatar
0 votes

xLSTM parallel computation - mismatch in dimensions

$$ Q \in R^{T \times d}, K \in R^{T \times d} \\ Q \cdot K^T \in R^{T \times d} \times R^{d \times T}\\ Q \cdot K^T \in R^{T \times T} $$ Which is exactly the dimension of $D$
Alberto's user avatar
  • 2,133
0 votes

What if the sum of word embedding and positional embedding becomes same for different words?

This would be incredibly difficult just due to the size of these embedding vectors and the fact that we are performing 2D rotations (many in parallel). The larger this embedding space the more the ...
naston's user avatar
  • 11
1 vote

Last linear layer of the decoder of a transformer

Well well, this text in the link posted in one of the answers above (https://www.tensorflow.org/text/tutorials/transformer) gives the answer to the question. Note: The model is optimized for ...
eddys's user avatar
  • 111
0 votes
Accepted

Understanding different methods of covariance parametrization

Parametrizing something is literally what the term is saying: "how do you put parameters so that you can end up having it via such parameters" For example, you can parametrize the covariance ...
Alberto's user avatar
  • 2,133
0 votes

What if the sum of word embedding and positional embedding becomes same for different words?

The positional embedding values in transformers are fixed from the start, they are calculated only once before the start of training (or stored/reused from years ago in memory). So there is only very ...
James's user avatar
  • 145
0 votes
Accepted

What does "position-wise" fully connected mean?

Position-Wise basically means that the architecture takes position of a given value in the vector into considiration, and changing position of a given value in the input will impact the output.
Saad El Kouari's user avatar
4 votes
Accepted

Is there a relationship between tokens and parameters in LLMs?

Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt) Consider that the context window that you see in the ...
Alberto's user avatar
  • 2,133
0 votes

How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?

They don't seem to be sharing any supplementary material with details on this, however as they state, convolution being independent from the input size can be applied to varying size input (see ...
Alberto's user avatar
  • 2,133

Top 50 recent answers are included