2

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs. The main part of the Vision Transformer are the self-attention layers. The architecture proposed in the paper An Image is Worth 16x16 Words treats each 16x16 as a word in the sentence. There is a convolutional layer (with kernel_size=16 and ...


2

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "...


1

Transformers, being a general-purpose sequence model can be used for Time-Series forecasting. There are some papers dedicated to the use of Transformer for time-series prediction and blogs. The main ingredient for the autoregression in predictions is the mask in Transformer encoder. When the next element is predicted, tokens in the sequence attend only to ...


Only top voted, non community-wiki answers of a minimum length are eligible