40 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
user avatar
30 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
user avatar
13 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
user avatar
  • 191
13 votes

How can I predict the next number in a non-obvious sequence?

This is a question of time series forecasting, since your numbers form a sequence. You may want to take a look at the "forecasting" tag at CrossValidated. If you have only 700 data points, ...
user avatar
11 votes

Where can I find the original paper that introduced RNNs?

The two tech reports below both call RNNs explicitly "recurrent net(work)s". Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations ...
user avatar
11 votes

What is a recurrent neural network?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
user avatar
  • 33.8k
10 votes
Accepted

Is LSTM a subcategory of RNN?

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that ...
user avatar
9 votes
Accepted

What is a recurrent neural network?

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
user avatar
  • 699
8 votes
Accepted

Which approaches could I use to create a simple chatbot using a neural network?

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that ...
user avatar
8 votes

How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...
user avatar
8 votes

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
user avatar
  • 266
8 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
user avatar
8 votes
Accepted

How can I predict the next number in a non-obvious sequence?

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious ...
user avatar
  • 870
7 votes

What is the fundamental difference between CNN and RNN?

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward ...
user avatar
  • 33.8k
7 votes

What is the fundamental difference between CNN and RNN?

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I ...
user avatar
  • 474
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
user avatar
  • 1,351
7 votes

How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
user avatar
  • 699
7 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
user avatar
  • 946
7 votes

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
user avatar
  • 1,715
7 votes
Accepted

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^...
user avatar
6 votes
Accepted

Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?

The approach you describe is called neuromorphic computing and it's quite a busy field. IBM's TrueNorth even has spiking neurons. The main problem with these projects is that nobody quite knows ...
user avatar
6 votes

What are the models that have the potential to replace neural networks in the near future?

This is going backwards, but it kind of follows the logic of the arguments. In terms of efficiency, I can see a few major problems with classical neural networks. Data collection and preprocessing ...
user avatar
6 votes

How can I predict the next number in a non-obvious sequence?

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/...
user avatar
  • 143
5 votes
Accepted

Do we have anything like accuracy and loss in RNN models?

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning. There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you ...
user avatar
  • 1,359
4 votes

Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?

If neurons and synapses can be implemented using transistors, I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, ...
user avatar
  • 1,007
4 votes
Accepted

How can I understand this statement about RNNs and hidden layers?

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman ...
user avatar
4 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
user avatar
  • 269
4 votes

Can we ever achieve hypercomputation using recurrent neural networks?

I presume the proof the OP is referring to can be found in this monograph by Hava Siegelmann? In his article 'The Myth of Hypercomputation', the eminent computer scientist Martin Davis explains (p8-9)...
user avatar
4 votes
Accepted

How to design a neural network when the number of inputs is variable?

The best option in your case would probably be zero-padding or padding up. This is simply zeroing out inputs for cases in which there is no data. It's done a lot on the borders of images for CNNs. ...
user avatar
4 votes
Accepted

Do convolutional neural networks also have recurrent connections?

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN. RNNs and CNNs are different architectures but they ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible