Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with transformer
Search options answers only
not deleted
user 38846
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
6
votes
Accepted
What is the difference between the positional encoding techniques of the Transformer and GPT?
Fixed encoding
In the original Transformer one uses a fixed map from the token position $i$ to the embedding vector added to the original embedding:
$$
\begin{aligned}
PE(\text{pos}, 2i) &= \sin(\text{ …
- 2,754
1
vote
Can I use the transformers for the prediction of historical data?
There are some papers dedicated to the use of Transformer for time-series prediction and blogs.
The main ingredient for the autoregression in predictions is the mask in Transformer encoder. …
- 2,754
5
votes
What is the Intermediate (dense) layer in between attention-output and encoder-output dense ...
Feedforward layer is an important part of the transformer architecture. … Transformer architecture, in addition to the self-attention layer, that aggregates information from the whole sequence and transforms each token due to the attention scores from the queries and values …
- 2,754
6
votes
Accepted
Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
Yes, they can handle sequences with arbitrary length sequence, but with some remarks.
In the paper Training data-efficient image transformers & distillation through attention authors train models in t …
- 2,754
4
votes
Accepted
What are the major layers in a Vision Transformer?
The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.
The main part of the Vision Transformer are the self-attention layers. … Each Transformer encoder is a standard Transformer block, consisting of the:
Multihead self-attention layer that transforms tokens into keys, queries and values
Feedforward layer acting on each token …
- 2,754