19 votes
Accepted

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
HLeb's user avatar
  • 579
13 votes

How can I predict the next number in a non-obvious sequence?

This is a question of time series forecasting, since your numbers form a sequence. You may want to take a look at the "forecasting" tag at CrossValidated. If you have only 700 data points, ...
Stephan Kolassa's user avatar
9 votes

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
Mathias Müller's user avatar
9 votes
Accepted

How can I predict the next number in a non-obvious sequence?

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious ...
serali's user avatar
  • 890
6 votes

How can I predict the next number in a non-obvious sequence?

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/...
Sanyou's user avatar
  • 165
4 votes
Accepted

What are the approaches to predict sequence of $\pi$ numbers?

Pseudo-random number generators are specifically defined to defeat any form of prediction via 'black box' observation. Certainly, some (e.g. linear congruential) have weaknesses, but you are unlikely ...
NietzscheanAI's user avatar
4 votes

How can I predict the next number in a non-obvious sequence?

Since you only have only 700 observations, I would not try a deep learning approach. I think it is very unlikely that any Deep Learning approach will learn a non-obvious relationship with that little ...
Akavall's user avatar
  • 216
4 votes
Accepted

Why do Transformers have a sequence limit at inference time?

Transformer models have limited sequence length at inference time because of positional embeddings. But there are workarounds. Self-attention in transformer does not distinguish the order of keys/...
user2907934's user avatar
3 votes

What is the proper way to process continuous sequence data, such as time-series, using the Transformer?

Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer: ...
Dylan Kerler's user avatar
3 votes

Can Reinforcement Learning be used to generate sequences?

One renowned example for the specified case is SeqGAN Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by ...
OmG's user avatar
  • 1,816
3 votes
Accepted

How to use LSTM to generate a paragraph

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That'...
jgleoj23's user avatar
  • 182
3 votes

How can I use machine learning to predict properties (such as the area) of simple polygons?

You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML. Anyway if you approximates these polygons as images you could maybe train ...
FourierFlux's user avatar
3 votes

In sequence-to-sequence, why is the output of the decoder used as its input?

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals: \begin{align*} p(w_1,w_2, \dots,w_n) &= \ p(w_1)*p(w_2|w_1) * \ ... \ *...
mshlis's user avatar
  • 2,359
3 votes

Why do we need both encoder and decoder in sequence to sequence prediction?

(Old question, I know...) It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often. Example for ...
Mathias Müller's user avatar
3 votes

Sequence Embedding using embedding layer: how does the network architecture influence it?

Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
Edoardo Guerriero's user avatar
3 votes
Accepted

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. ...
Victor Björkgren's user avatar
2 votes

Can the recurrent neural network's input come from a short-time Fourier transform?

Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is ...
Neil Slater's user avatar
  • 32.1k
2 votes

How to use LSTM to generate a paragraph

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens ...
Clement's user avatar
  • 1,745
2 votes

Can the decoder in a transformer model be parallelized like the encoder?

Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. ...
Ben's user avatar
  • 81
2 votes
Accepted

How do LSTM or GRU gates learn to specialize in their desired tasks?

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer. Looking at the forget gate, you can see that it has the ability to manipulate ...
Recessive's user avatar
  • 1,396
2 votes

Is there any reason for giving an index to a token based on its frequency in the text?

The main reason is given in the next sentence, i.e. after crating a corpus we want to know which words are the most frequent and which one are the most rare. All together, rare and frequent words are ...
Edoardo Guerriero's user avatar
2 votes

Is there any reason for giving an index to a token based on its frequency in the text?

In NLP, word types (not tokens) are often represented by numbers, as they are easier to process: They all take the same space, so random access is a lot easier than with variable length strings, and ...
Oliver Mason's user avatar
  • 5,387
2 votes
Accepted

Difference between dot product attention and "matrix attention"

The key difference between the attention mechanisms used in "Learning To Align And Translate" and "Attention Is All You Need" lies in the way that the similarity between the query ...
hH1sG0n3's user avatar
  • 211
2 votes
Accepted

How is the padding mask incorporated in the attention formula?

Entries of an attention mask are typically either $0$ or $-\infty$. So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$. The issue with entrywise multiplication with a binary ...
Venna Banana's user avatar
2 votes

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd ...
programjames's user avatar
1 vote

Why do Transformers have a sequence limit at inference time?

To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length. However, the ...
Avatrin's user avatar
  • 486
1 vote

Normalization of possibly not fully representative data

A friend of mine answered on this question in different social media on different language, I'll post his answer here: 1. scaler should be saved in this case. You do fit_transform in the example the ...
banderlog013's user avatar
1 vote
Accepted

Can we use transformers for audio classification tasks?

Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech ...
user3667125's user avatar
  • 1,570
1 vote
Accepted

Mining repeated subsequences in a given sequence

You can do this similar to the BIDE approach. It can be done like this: ...
jottbe's user avatar
  • 176
1 vote
Accepted

Number of LSTM layers needed to learn a certain number of sequences

From my personal experience, the units hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster. ...
Dan D.'s user avatar
  • 1,283

Only top scored, non community-wiki answers of a minimum length are eligible