7

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...


6

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase). What exactly is parallelized? Also, it's worth ...


3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...


3

You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML. Anyway if you approximates these polygons as images you could maybe train a CNN. Look at the image classification networks which provide bounding boxes.


3

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals: $\begin{align*} p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\ =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i}) \end{align*} $ This can be sampled by samping each of the conditional is ascending ...


2

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer. Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (...


2

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as ...


2

(Old question, I know...) It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often. Example for Sequence-to-sequence without two RNNs To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-...


1

Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech Recognition with the Transformer Model), and text-to-speech (Neural Speech Synthesis with Transformer Network).


1

You can do this similar to the BIDE approach. It can be done like this: class TreeNode: def __init__(self, element, depth, count=0, parent=None): self.count= count self.element= element self.depth= depth self.subnodes= dict() self.parent= parent def __repr__(self): return f'{self.__class__....


1

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference. I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short. Either way, when a model is trained a certain way,...


1

When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens. OpenAI did some ...


1

In computational learning theory, the VC dimension is a formal measure of the capacity of a model. The VC dimension is defined in terms of the concept of shattering, so have a look at the related Wikipedia article, which briefly describes the fundamental concept of shattering. See also my answer to the question How to estimate the capacity of a neural ...


1

If you give it a name, it will probably be almost perfect model, as the number of hidden units are definitely enough to store the raw data of a name. However as a neural network it will still not be perfect. There may also be serious overfitting if you do this. A Seq2Seq model have way more parameters than necessary to just "remembers" all the possible names,...


1

He likely found this to a be a best practice to avoid over fitting, with a small data set if you only use small and easy to learn (less words -> less degrees of freedom) sequences then you open your model to the risk of over fitting that data set where as on a large data set that has alot more total information you can train on small sequences without being ...


1

Shouldn't the hidden state of the RNN from the previous timestamps be enough? It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras: Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training ...


1

In section 2.1 of the paper, the authors state that the goal of character-level language model is to estimate the following joint probability distribution $$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$ which is a joint probability distribution of all characters of a sequence of $T+1$ characters, where $\...


1

Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. This typically produces worse accuracy scores because there are interdependencies within the text being produced - a model translating "thank you" could ...


1

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM. Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to ...


Only top voted, non community-wiki answers of a minimum length are eligible