Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
66 votes
4 answers
119k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
Stephen Johnson's user avatar
35 votes
2 answers
19k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
chessprogrammer's user avatar
71 votes
4 answers
96k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
DRV's user avatar
  • 1,673
13 votes
5 answers
8k views

What is the fundamental difference between CNN and RNN?

What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?
Pradeep BV's user avatar
13 votes
2 answers
3k views

What is a recurrent neural network?

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular (or ...
olinarr's user avatar
  • 755
7 votes
1 answer
27k views

What is the difference between LSTM and RNN?

What is the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
Mao76's user avatar
  • 73
3 votes
1 answer
981 views

What is teacher forcing?

In the paper Neural Programmer-Interpreters, the authors use the teacher forcing technique, but what exactly is it?
nbro's user avatar
  • 40.2k
3 votes
1 answer
492 views

What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper?

Researchers at Stanford University released, in 2012, the paper Financial Market Time Series Prediction with Recurrent Neural Networks. It goes on to discuss how they used echo state networks to ...
BleedObsidian's user avatar
19 votes
4 answers
27k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
Ahsan Tarique's user avatar
4 votes
2 answers
984 views

Are there neural networks that accept graphs or trees as inputs?

As far I know, the RNN accepts a sequence as input and can produce as a sequence as output. Are there neural networks that accept graphs or trees as inputs, so that to represent the relationships ...
user8426627's user avatar
3 votes
0 answers
96 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
3 votes
2 answers
5k views

Spam Detection using Recurrent Neural Networks [closed]

I am working on this code for spam detection using recurrent neural networks. Question 1. I am wondering whether this field (using RNNs for email spam detection) worths more researches or it is a ...
Hossein's user avatar
  • 177
2 votes
1 answer
4k views

What are some examples of LSTM architectures?

I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" ...
thegreatjedi's user avatar
1 vote
0 answers
72 views

Is the capability of RNN more than the capability of MLP?

Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
hanugm's user avatar
  • 3,792
1 vote
0 answers
126 views

Is a true RNN auto encoder possible with Keras/TF

I want to get some encodings for temporal data (with a highly varying number of timesteps). The dataset is of the format: ...
Tobi Akinyemi's user avatar
1 vote
1 answer
282 views

Do you need to store prevous values of weights and layers on recurrent layer while BPTT?

The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like ...
user8426627's user avatar
0 votes
1 answer
1k views

How to process data in a data stream for a LSTM

How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes? Training phase: I have trained a LSTM to predict a class out of a ...
MScott's user avatar
  • 445
0 votes
1 answer
54 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
The Pointer's user avatar
-4 votes
2 answers
864 views

What are the approaches to predict sequence of $\pi$ numbers? [closed]

Given a list of fixed numbers from a mathematical constant, such as $\pi$, is it is possible to train AI to attempt to predict the next numbers of this constant? Which AI or neural network would be ...
kenorb's user avatar
  • 10.5k