Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
62 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
Edoardo Guerriero's user avatar
37 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
Thomas Wagenaar's user avatar
31 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
tsu's user avatar
  • 471
24 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
David's user avatar
  • 4,910
15 votes

Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?

There is huge difference between what is happening with the information during training and during inference and one can not be used for the other. Let me start with an analogy to the human brain (...
Broele's user avatar
  • 561
14 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
Eka's user avatar
  • 1,066
13 votes

Where can I find the original paper that introduced RNNs?

The two tech reports below both call RNNs explicitly "recurrent net(work)s". Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations ...
David Nemeskey's user avatar
13 votes

How can I predict the next number in a non-obvious sequence?

This is a question of time series forecasting, since your numbers form a sequence. You may want to take a look at the "forecasting" tag at CrossValidated. If you have only 700 data points, ...
Stephan Kolassa's user avatar
11 votes

What is a recurrent neural network?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
nbro's user avatar
  • 40.8k
11 votes

Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?

They are not "learning" during inference at all. Learning is the process of updating the weights of the model (to lower loss). This does not happen during inference. The model weights stay ...
shatz's user avatar
  • 144
10 votes

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
olix20's user avatar
  • 286
10 votes
Accepted

Is LSTM a subcategory of RNN?

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that ...
Neil Slater's user avatar
  • 32.4k
9 votes

How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...
Maheshwar Ligade's user avatar
9 votes
Accepted

What is a recurrent neural network?

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
naive's user avatar
  • 699
9 votes
Accepted

What is the difference between LSTM and RNN?

RNNs have recurrent connections and/or layers You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For ...
nbro's user avatar
  • 40.8k
9 votes
Accepted

How can I predict the next number in a non-obvious sequence?

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious ...
serali's user avatar
  • 890
8 votes
Accepted

Which approaches could I use to create a simple chatbot using a neural network?

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that ...
BlindKungFuMaster's user avatar
8 votes

What is the fundamental difference between CNN and RNN?

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward ...
nbro's user avatar
  • 40.8k
7 votes

What is the fundamental difference between CNN and RNN?

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I ...
pshlady's user avatar
  • 484
7 votes

How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
naive's user avatar
  • 699
7 votes
Accepted

Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?

The approach you describe is called neuromorphic computing and it's quite a busy field. IBM's TrueNorth even has spiking neurons. The main problem with these projects is that nobody quite knows ...
BlindKungFuMaster's user avatar
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
Dawny33's user avatar
  • 1,371
7 votes

Where can I find the original paper that introduced RNNs?

Hopfield networks, a special case of RNNs, were first proposed in 1982: https://www.pnas.org/content/79/8/2554 Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found ...
AlDante's user avatar
  • 216
7 votes

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
Clement's user avatar
  • 1,745
7 votes

How can Transformers handle arbitrary length input?

The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum ...
David Hoelzer's user avatar
7 votes
Accepted

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^...
David's user avatar
  • 4,910
6 votes

What are the models that have the potential to replace neural networks in the near future?

This is going backwards, but it kind of follows the logic of the arguments. In terms of efficiency, I can see a few major problems with classical neural networks. Data collection and preprocessing ...
cantordust's user avatar
6 votes

How can I predict the next number in a non-obvious sequence?

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/...
Sanyou's user avatar
  • 165
6 votes
Accepted

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

No, ResNet were not introduced to solve vanishing gradients, citing from the paper: An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which ...
Alberto's user avatar
  • 2,133
5 votes

Is there a rigorous proof for finding Hopfield minima?

See the paper On the Convergence Properties of the Hopfield Model (1990), by Jehoshua Bruck. In the first section of the paper, J. Bruck describes the Hopfield network (popularized by J. J. Hopfield ...
nbro's user avatar
  • 40.8k

Only top scored, non community-wiki answers of a minimum length are eligible