Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
34
votes
4answers
53k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
17
votes
1answer
384 views

Could a Boltzmann machine store more patterns than a Hopfield net?

This is from a closed beta for AI, with this question being posted by user number 47. All credit to them. According to Wikipedia, Boltzmann machines can be seen as the stochastic, generative ...
14
votes
2answers
259 views

How do I decide the optimal number of layers for a neural network?

How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
12
votes
2answers
2k views

What is a Recurrent Neural Network?

Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular NNs?
10
votes
5answers
6k views

What is the fundamental difference between CNN and RNN?

What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?
10
votes
2answers
4k views

How to train a chatbot

I wanted to started experimenting with neural network and as a toy problem I wished to train one to chat, i.e. implement a chatting bot like cleverbot. Not that clever anyway. I looked around for ...
10
votes
4answers
781 views

What are the models that have the potential to replace neural networks in the near future?

Are there possible models that have the potential to replace neural networks in the near future? And do we even need that? What is the worst thing about using neural networks in terms of efficiency?
9
votes
4answers
724 views

Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?

If neurons and synapses can be implemented using transistors, what prevents us from creating arbitrarily large neural networks using the same methods with which GPUs are made? In essence, we have ...
8
votes
2answers
750 views

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs ...
7
votes
2answers
4k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
7
votes
2answers
2k views

Structure of LSTM RNNs

I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you! For ...
7
votes
1answer
153 views

Neural network design when amount of input neurons vary

I'm looking to design a neural network that can predict which runner wins in a sports game, where the amount of runners varies between 2-10. In each case, specific data about the individual runners ...
6
votes
3answers
554 views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
6
votes
1answer
1k views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
6
votes
3answers
81 views

In sequence-to-sequence, why is the output of the decoder used as its input?

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence: Why is the output, ...
5
votes
1answer
101 views

Are neurons instantly feed forward when input arrives?

Let's say I have a neural network with 5 layers, including the input and output layer. Each layer has 5 nodes. Assume the layers are fully connected, but the 3rd node in the 2nd layer is connected to ...
5
votes
1answer
69 views

Why do small datasets require more samples, while big datasets require fewer samples in negative sampling?

In the deep learning specialization course by Andrew Ng, in the video Sequence Models (minute 4:13), he says that in negative sampling we have to choose a sample of words from the corpus to train ...
5
votes
1answer
532 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
5
votes
2answers
3k views

Spam Detection using Recurrent Neural Networks

I am working on this code for spam detection using recurrent neural networks. Question 1. I am wondering whether this field (using RNNs for email spam detection) worths more researches or it is a ...
5
votes
1answer
489 views

What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, ...
4
votes
3answers
109 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
4
votes
1answer
1k views

How can I understand this statement about RNNs and hidden layers?

In the lecture, there was a statement: Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing. I understand ...
4
votes
2answers
79 views

Why are GRU and LSTM better than standard RNNs?

It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.
4
votes
3answers
147 views

What kinds of systems have so far failed to be modeled via supervised artificial network training?

Artificial networks model systems with a set of inputs and outputs and expected behavior. To train a network for modeling such systems, hundreds, thousands, or millions of example inputs-output pairs ...
4
votes
1answer
44 views

Training an RNN to answer simple quesitons

I would like to train an RNN to follow the sentences: "Would you like some cheese"? with "Yes, I would like some cheese." So whenever the template "Would you like some ____?" appears then RNN ...
4
votes
1answer
450 views

Over- and underestimations of the lowest and highest values in LSTM network

I'm training a LSTM network with multiple inputs and several LSTM layers in order to setup a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation on the ...
4
votes
0answers
60 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
4
votes
0answers
82 views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
4
votes
0answers
54 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
3
votes
2answers
104 views

What is the relation between Convolutional Neural Networks and Recurrent Neural Networks?

I asked my self this simple question while reading "Comment Abuse Classification with Deep Learning" by Chu and Jue. Indeed, they say at the end of the that It is clear that RNNs, specifically ...
3
votes
2answers
150 views

How do layers in an artificial neural network transform inputs to outputs?

To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, ...
3
votes
1answer
714 views

Fourier Transform inputs (Frequency) for RNN

Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.
3
votes
1answer
424 views

What is a state in a recurrent neural network?

I am Reading "Supervised Sequence Labelling with Recurrent Neural Networks" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations. Specifically, what I ...
3
votes
1answer
51 views

Why RNNs often use just one hidden layer?

Did I get it right, that RNNs most often have just one hidden neuron layer? Is there a reason for that? Will RNNs with several hidden layers in each cell work worse? Thank you!!
3
votes
1answer
121 views

Is it possible to use an RNN to predict a feature that is not an input feature?

I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now. Let's say I have a set of input features, generated every second. Corresponding with ...
3
votes
1answer
82 views

Issue at training simple RNN for word generation

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. This is done with ...
3
votes
2answers
71 views

Conferences for Human Activity Recognition

What are some conferences for publishing papers on Deep Learning for Human Activity recognition? Do any of the major conferences have specific tracks for Human Activity Recognition?
3
votes
1answer
335 views

Additive Attention in Convolutional Networks

Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures ...
3
votes
1answer
95 views

Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?

I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG. While reading ...
3
votes
2answers
198 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
3
votes
2answers
291 views

Is there an alternative to RNNs that doesn't require knowing input history?

In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes. This doesn't seem like a realistic picture of the brain since this would ...
3
votes
0answers
17 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
3
votes
0answers
28 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
3
votes
0answers
16 views

How to back propagate for implementation of Sequence-to-Sequence with Multi Decoders

I am proposing a modified version of Sequence-to-Sequence model with dual decoders. The problem that I am trying to solve is Neural Machine Translation into two languages at once. This is the ...
3
votes
0answers
49 views

Ideas on a network that can translate image differences into motor commands?

I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
3
votes
1answer
56 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
3
votes
0answers
122 views

How does bidirectional encoding allow the predicted word to indirectly “see itself”?

Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
3
votes
0answers
103 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
3
votes
0answers
107 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
3
votes
1answer
1k views

Seq2Seq dialogs predicts only most common words like `you` after couple of epoches

I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus. My work based on the following papers (but currently I'm not implemented Attention yet): Sequence to Sequence ...