61 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
Edoardo Guerriero's user avatar
36 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
Thomas Wagenaar's user avatar
31 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
tsu's user avatar
  • 471
19 votes
Accepted

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
HLeb's user avatar
  • 579
17 votes
Accepted

How does LSTM in deep reinforcement learning differ from experience replay?

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one ...
Neil Slater's user avatar
  • 31.7k
14 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
Eka's user avatar
  • 1,066
13 votes

Where can I find the original paper that introduced RNNs?

The two tech reports below both call RNNs explicitly "recurrent net(work)s". Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations ...
David Nemeskey's user avatar
10 votes
Accepted

Is LSTM a subcategory of RNN?

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that ...
Neil Slater's user avatar
  • 31.7k
9 votes

Can LSTM neural networks be sped up by a GPU?

From Nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and ...
pasaba por aqui's user avatar
9 votes

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
Mathias Müller's user avatar
9 votes

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
olix20's user avatar
  • 276
8 votes

How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...
Maheshwar Ligade's user avatar
8 votes
Accepted

Which approaches could I use to create a simple chatbot using a neural network?

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that ...
BlindKungFuMaster's user avatar
8 votes
Accepted

What is the difference between LSTM and RNN?

RNNs have recurrent connections and/or layers You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For ...
nbro's user avatar
  • 40.2k
7 votes

How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
naive's user avatar
  • 699
7 votes

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
Clement's user avatar
  • 1,735
7 votes

What are pros and cons of Bi-LSTM as compared to LSTM?

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to ...
Edoardo Guerriero's user avatar
6 votes
Accepted

Can LSTM neural networks be sped up by a GPU?

I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm. They are very fast. The normal LSTM cells are faster on CPU than on GPU.
Dieshe's user avatar
  • 289
6 votes

Where can I find the original paper that introduced RNNs?

Hopfield networks, a special case of RNNs, were first proposed in 1982: https://www.pnas.org/content/79/8/2554 Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found ...
AlDante's user avatar
  • 206
6 votes

Should I choose a model with the smallest loss or highest accuracy?

You should choose the model A. The loss is just a differentiable proxy for accuracy. That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine ...
ssegvic's user avatar
  • 499
6 votes

Is the LSTM component a neuron or a layer?

The diagram you show works at least partially for describing both individual neurons and layers of those neurons. However, the "incoming" data lines on the left represent all inputs under ...
Neil Slater's user avatar
  • 31.7k
5 votes
Accepted

What is the difference between ConvLSTM and CNN LSTM?

After doing a bit of research I found that the LSTM whose gates perform convolutions is called ConvLSTM. The term CNN LSTM is loose and may mean stacking up LSTM on top of CNN for tasks like video ...
user239457's user avatar
5 votes
Accepted

How to shorten the development time of a neural network?

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
SmallChess's user avatar
  • 1,411
5 votes
Accepted

How does the forget layer of an LSTM work?

TL;DR Here is a beautiful explanation with diagrams: source. To address: the cell state is essentially long term memory embedding (correct me if I'm wrong) The embedding can be long or short term ...
respectful's user avatar
  • 1,106
5 votes

How can Transformers handle arbitrary length input?

The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum ...
David Hoelzer's user avatar
5 votes

Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Yes an LSTM can use any of these. There are no hard rules of which to use. That is why they all exist. Some rules of thumb are: Relu is the cheapest computationally. Almost always worth trying first. ...
chessprogrammer's user avatar
5 votes
Accepted

Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Yes, you can use ReLU or LeakyReLU in an LSTM model. There aren't hard rules for choosing activation functions. Run your model with each activation function and pick the best performing one. See the ...
Brian O'Donnell's user avatar
4 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
Dieshe's user avatar
  • 289
4 votes
Accepted

Why are GRU and LSTM better than standard RNNs?

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual ...
Simbarashe Timothy Motsi's user avatar
4 votes

Where can I find the original paper that introduced RNNs?

Warren McCulloch and Walter Pitts talk about recurrent neural nets in their paper McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical ...
Vladislav Gladkikh's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible