# Tag Info

Accepted

### Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
• 4,283

• 1,260
Accepted

### What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix \$\|\text{sentence length}\| \times \|\...
• 1,260

### In layman terms, what does "attention" do in a transformer?

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
• 4,283
Accepted

### What are the major layers in a Vision Transformer?

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs. The main part of the Vision Transformer are the self-attention layers. The ...
Accepted

### Any comparison between transformer and RNN+Attention on the same dataset?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
• 1,683

### What is the purpose of Decoder mask (triangular mask) in Transformer?

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism. Encoder: Both runtime or training: the encoder will ...
• 210

### Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer. The feed-forward networks as ...
• 81
Accepted

### What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
Accepted

### Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
• 1,260

### What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
• 1,503
Accepted

### How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

In short, repetition with feedback. You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work ...
• 24.7k
Accepted

### Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
• 194
Accepted

### Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?

"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates ...
• 5,167

### What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

Feedforward layer is an important part of the transformer architecture. Transformer architecture, in addition to the self-attention layer, that aggregates ...