4

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\text{vocab}\|$ (right before taking the argmax to find the token to output), and the $\|\text{sentence length}\|$-length vector of token IDs as the true label.


4

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...


3

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio. ...


3

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...


3

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $...


3

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\ Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$ Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices,...


3

"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates from principal component analysis, where uncorrelated variation would be along orthogonal axes. Or, in the words of the Wikipedia article on orthogonality applied ...


2

The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of mask to -inf (when it should be $0$). Also, just an FYI, although you haven't focused on evaluation/prediction yet, I will explain the evaluation/prediction here as well for completeness, since it works so differently than ...


2

There are a few more books that were published after 2016 that cover some of the topics you are interested in. I've not read any of them, so I don't really know whether they are good or not, but I try to summarise if they cover some of the topics you may be interested in. Deep Learning with Python (2017), by Francois Chollet (author of the initial Keras ...


2

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & Attention (all three occurs in the index). But read also Pitrat's last book: Artificial Beings: The Conscience of a Conscious Machine - it does cover machine learning (but not in the "deep learning" sense) but was ...


2

In short, repetition with feedback. You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work around this when desired input and output is more variable. The most common approaches are: Padding: Give the ML model capacity to cope with the largest expected ...


2

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention head is the same. For attention head $i$: $Q_i(x) = x W_i^Q \\ K_i(x) = x W_i^K \\ V_i(x) = x W_i^V \\ attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\...


2

Read the paper. It tells you. (page 3) Bit memory. Similar to the task proposed by Miconi et al. (2018), we consider a bit memory task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the model is tasked with producing the ...


2

You simply split the sequence into smaller sequences; while there are some long-distance dependencies in language, that is generally not a problem for this. A sentence would typically be short enough, and very long sentences are composed of shorter clauses which would form independent units (albeit connected with each other).


2

Edit I just noticed that the model you are referring to is built very differently than the transformer from Attention is All You Need since it only uses one half of the architecture. Thus my answer below is not be complete. I thus have to add the following: (The final two paragraphs still apply as they are, though) The Keras model is quite weird, and while ...


2

Looking at the paper, it seems to me that they are not using orthogonal in a literal, mathematics (or geometric) sense. Instead, I read that as two things (especially since the word "ablation" appears later in the sentence): They are attempting to use lots of fancy words They are simply indicating that these changes are separate from and have no ...


2

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "...


1

An Image is Worth 16X16 Words: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE A Transformer consists of alternating layers of multiheaded self-attention. The Transformer Paper adapts a NLP architecture for making Image Classification. For that, it first need to tokenize the image (like a piece of text). The tokenization is done by splitting the image into fixed-...


1

An RNN processes words one by one. For example on the sentence "man eats dog", it will: Fully process "man", producing an output $y_1$ and hidden units $h_1$. Fully process "eats", now using also the previous output and/or hidden units. Finally process "dog", again using the previous output and/or hidden units $y_2$ ...


1

There is not one answer to this question, but one could argue that transformers heavily rely on transforming each input into latent subspaces of queries, keys and values in order to generate attention score a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of ...


1

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments "masked". The model is asked to reconstruct the masked parts. To ...


1

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot taking a dialogue context and producing a response. In these cases, there are qualitative differences between the inputs and outputs so that it makes sense to ...


1

There is an implementation of the paper ("Adversarial Sparse Transformer for Time Series Forecasting"), in Python using Pytorch, here. Although it has the training and evaluation functionality implemented, it appears to be lacking a function for running a prediction. Maybe you can fork it and extend it. UPDATE There is also a paper, "Informer:...


1

Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech Recognition with the Transformer Model), and text-to-speech (Neural Speech Synthesis with Transformer Network).


1

Yes, there is. You can try Spacy. Here you go. import spacy from spacytextblob.spacytextblob import SpacyTextBlob nlp = spacy.load('en_core_web_sm') spacy_text_blob = SpacyTextBlob() nlp.add_pipe(spacy_text_blob) text = "i'm good" doc = nlp(text) print(doc._.sentiment.polarity) # 0.7 text = "i'm bad" doc = nlp(text) print(doc._....


1

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight ...


1

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. The embeddings are trained from scratch.


1

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples: ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, ...


1

I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation. The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture. $W$ has dimension $d_{model} \times d_{model}$, which means that we are adding $d_{...


1

The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training. This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder: class Decoder(nn.Module): "...


Only top voted, non community-wiki answers of a minimum length are eligible