19
votes
Accepted
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the
encoder?
Generally NO:
Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
- 579
13
votes
How can I predict the next number in a non-obvious sequence?
This is a question of time series forecasting, since your numbers form a sequence.
You may want to take a look at the "forecasting" tag at CrossValidated.
If you have only 700 data points, ...
- 231
9
votes
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
- 361
9
votes
Accepted
How can I predict the next number in a non-obvious sequence?
As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious ...
- 890
6
votes
How can I predict the next number in a non-obvious sequence?
I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/...
- 165
4
votes
Accepted
What are the approaches to predict sequence of $\pi$ numbers?
Pseudo-random number generators are specifically defined to defeat any form of prediction via 'black box' observation. Certainly, some (e.g. linear congruential) have weaknesses, but you are unlikely ...
- 7,276
4
votes
How can I predict the next number in a non-obvious sequence?
Since you only have only 700 observations, I would not try a deep learning approach. I think it is very unlikely that any Deep Learning approach will learn a non-obvious relationship with that little ...
- 216
4
votes
Accepted
Why do Transformers have a sequence limit at inference time?
Transformer models have limited sequence length at inference time
because of positional embeddings. But there are workarounds.
Self-attention in transformer does not distinguish the order of keys/...
- 76
3
votes
What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:
...
- 283
3
votes
Can Reinforcement Learning be used to generate sequences?
One renowned example for the specified case is SeqGAN
Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by ...
- 1,826
3
votes
Accepted
How to use LSTM to generate a paragraph
Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That'...
- 182
3
votes
How can I use machine learning to predict properties (such as the area) of simple polygons?
You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML.
Anyway if you approximates these polygons as images you could maybe train ...
- 863
3
votes
In sequence-to-sequence, why is the output of the decoder used as its input?
In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:
\begin{align*}
p(w_1,w_2, \dots,w_n) &= \ p(w_1)*p(w_2|w_1) * \ ... \ *...
- 2,379
3
votes
Why do we need both encoder and decoder in sequence to sequence prediction?
(Old question, I know...)
It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.
Example for ...
- 361
3
votes
Sequence Embedding using embedding layer: how does the network architecture influence it?
Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
- 5,378
3
votes
Accepted
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. ...
2
votes
Can the recurrent neural network's input come from a short-time Fourier transform?
Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is ...
- 32.9k
2
votes
How to use LSTM to generate a paragraph
As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens ...
- 1,745
2
votes
Can the decoder in a transformer model be parallelized like the encoder?
Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. ...
- 91
2
votes
Accepted
How do LSTM or GRU gates learn to specialize in their desired tasks?
It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer.
Looking at the forget gate, you can see that it has the ability to manipulate ...
- 1,406
2
votes
Is there any reason for giving an index to a token based on its frequency in the text?
The main reason is given in the next sentence, i.e. after crating a corpus we want to know which words are the most frequent and which one are the most rare.
All together, rare and frequent words are ...
- 5,378
2
votes
Is there any reason for giving an index to a token based on its frequency in the text?
In NLP, word types (not tokens) are often represented by numbers, as they are easier to process: They all take the same space, so random access is a lot easier than with variable length strings, and ...
- 5,397
2
votes
Accepted
Difference between dot product attention and "matrix attention"
The key difference between the attention mechanisms used in "Learning To Align And Translate" and "Attention Is All You Need" lies in the way that the similarity between the query ...
- 221
2
votes
Accepted
How is the padding mask incorporated in the attention formula?
Entries of an attention mask are typically either $0$ or $-\infty$.
So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$.
The issue with entrywise multiplication with a binary ...
- 406
2
votes
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd ...
- 166
1
vote
Why do Transformers have a sequence limit at inference time?
To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length.
However, the ...
- 506
1
vote
Normalization of possibly not fully representative data
A friend of mine answered on this question in different social media on different language, I'll post his answer here:
1. scaler should be saved in this case.
You do fit_transform in the example the ...
- 111
1
vote
Accepted
Can we use transformers for audio classification tasks?
Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech ...
- 1,660
1
vote
Accepted
Mining repeated subsequences in a given sequence
You can do this similar to the BIDE approach. It can be done like this:
...
- 176
1
vote
Accepted
Number of LSTM layers needed to learn a certain number of sequences
From my personal experience, the units hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.
...
- 1,293
Only top scored, non community-wiki answers of a minimum length are eligible