Questions tagged [recurrent-neural-networks]

Use for questions about Recurrent Neural Networks

Filter by
Sorted by
Tagged with
28
votes
4answers
38k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
16
votes
1answer
341 views

Could a Boltzmann machine store more patterns than a Hopfield net?

This is from a closed beta for AI, with this question being posted by user number 47. All credit to them. According to Wikipedia, Boltzmann machines can be seen as the stochastic, generative ...
12
votes
2answers
2k views

What is a Recurrent Neural Network?

Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular NNs?
9
votes
2answers
4k views

How to train a chatbot

I wanted to started experimenting with neural network and as a toy problem I wished to train one to chat, i.e. implement a chatting bot like cleverbot. Not that clever anyway. I looked around for ...
9
votes
4answers
568 views

Beyond neural networks?

Are there possible algorithms that have the potential to replace neural nets in the near future? And do we need that? What is the worst thing of using neural networks in terms of efficiency?
8
votes
4answers
661 views

Arbitrarily big neural network

If neurons and synapses can be implemented using transistors, what prevents us from creating arbitrarily large neural networks using the same methods with which GPUs are made? In essence, we have ...
7
votes
4answers
4k views

What is the fundamental difference between CNN and RNN?

What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?
7
votes
1answer
131 views

Neural network design when amount of input neurons vary

I'm looking to design a neural network that can predict which runner wins in a sports game, where the amount of runners varies between 2-10. In each case, specific data about the individual runners ...
6
votes
3answers
71 views

In sequence-to-sequence, why is the output of the decoder used as its input?

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence: Why is the output, ...
5
votes
1answer
2k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
5
votes
2answers
913 views

Structure of LSTM RNNs

I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you! For ...
5
votes
2answers
2k views

Spam Detection using Recurrent Neural Networks

I am working on this code for spam detection using recurrent neural networks. Question 1. I am wondering whether this field (using RNNs for email spam detection) worths more researches or it is a ...
4
votes
2answers
90 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
4
votes
1answer
685 views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
4
votes
1answer
926 views

Recurrent neural networks with hidden layer

In the lecture, there was a statement: "Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing." I understand ...
4
votes
1answer
94 views

Are Neurons instantly feed forward when input arrives?

Lets say I have a Neural Network with 5 layers, including input and output layer. Each Layer has 5 nodes. Assume the Layers are fully connected, but the 3rd Node in the 2nd Layer is connected to the ...
4
votes
3answers
145 views

What kinds of systems have so far failed to be modeled via supervised artificial network training?

Artificial networks model systems with a set of inputs and outputs and expected behavior. To train a network for modeling such systems, hundreds, thousands, or millions of example inputs-output pairs ...
4
votes
1answer
347 views

Over- and underestimations of the lowest and highest values in LSTM network

I'm training a LSTM network with multiple inputs and several LSTM layers in order to setup a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation on the ...
4
votes
1answer
30 views

dimensions of hidden layer and cell state layer in LSTM

I was following some examples to get familiar with tensorflow LSTM related api, but noticed that all LSTM initialization functions require only num_units parameter which denotes number of hidden units ...
3
votes
2answers
99 views

What is the relation between Convolutional Neural Networks and Recurrent Neural Networks?

I asked my self this simple question while reading "Comment Abuse Classification with Deep Learning" by Chu and Jue. Indeed, they say at the end of the that It is clear that RNNs, specifically ...
3
votes
2answers
123 views

How do layers in an artificial neural network transform inputs to outputs?

To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, ...
3
votes
1answer
488 views

Fourier Transform inputs (Frequency) for RNN

Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.
3
votes
1answer
195 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
3
votes
1answer
60 views

Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?

I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG. While reading ...
3
votes
2answers
154 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
3
votes
2answers
218 views

Is there an alternative to RNNs that doesn't require knowing input history?

In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes. This doesn't seem like a realistic picture of the brain since this would ...
3
votes
0answers
32 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
3
votes
0answers
94 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
3
votes
2answers
295 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. And I specifically mean language model in its standard sense. I ...
3
votes
1answer
858 views

Seq2Seq dialogs predicts only most common words like `you` after couple of epoches

I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus. My work based on the following papers (but currently I'm not implemented Attention yet): Sequence to Sequence ...
3
votes
0answers
115 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in http://karpathy.github.io/2015/05/21/rnn-effectiveness/) that when training RNN's on text for something like language modeling, the text is usually featurized character-by-...
2
votes
2answers
232 views

Can we ever achieve hypercomputation using recurrent neural networks?

It is proved that a recurrent neural net with rational weights can be a super-Turing machine. Can we achieve this in practice ?
2
votes
1answer
167 views

Active Learning and RNN

We have a series of data which want to label part of each series. As we do not have any training data, we try using Active Learning as a solution. But, the problem is our classifier is something like ...
2
votes
1answer
83 views

Is it possible to use an RNN to predict a feature that is not an input feature?

I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now. Let's say I have a set of input features, generated every second. Corresponding with ...
2
votes
1answer
198 views

What is a state in a recurrent neural network?

I am Reading "Supervised Sequence Labelling with Recurrent Neural Networks" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations. Specifically, what I ...
2
votes
2answers
35 views

Conferences for Human Activity Recognition

What are some conferences for publishing papers on Deep Learning for Human Activity recognition? Do any of the major conferences have specific tracks for Human Activity Recognition?
2
votes
1answer
40 views

How can I keep context in my chatbot

I have created a chatbot by Keras based on movie dialog. I used RNN more specifically GRU . My bot can reply well. But the problem is , it can't hold the context . As an example if I say ...
2
votes
1answer
26 views

Changes in flow detection neural network?

Do you have any advice, what architecture of neural network is the best for following task? Let input be some (complex function), the neural network gains a flow of its values, so I guess there will ...
2
votes
1answer
38 views

Why are GRU and LSTM better than old types of RNN?

Seems older RNNs have a limitation for their use cases and have been outperformed by other architectures for specific tasks e.g GRUS and CNNs
2
votes
1answer
54 views

Issue at training simple RNN for word generation

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. This is done with ...
2
votes
1answer
32 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
2
votes
0answers
40 views

How can my Neural Network categorize message strings?

Abstract I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. ...
2
votes
1answer
187 views

Additive Attention in Convolutional Networks

Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures ...
2
votes
0answers
85 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
2
votes
0answers
95 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
2
votes
0answers
91 views

Combine two embeddding inputs to increase more performance in LSTM model

The situation I encountered here is that I have two inputs(for instance, image embedding, etc.) into the first lstm of a series of lstms to predict the next word to generate sentence(from the second ...
2
votes
0answers
35 views

Detecting symmetry in small images with RNN

My network works on 32x32 normalized (translationally) but noisy images. Its task it to determine whether image has simple symmetry (horizontal/vertical). It needs to be reasonably robust to rotation (...
2
votes
0answers
45 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
2
votes
0answers
49 views

Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?

I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron ...
2
votes
0answers
64 views

seq2seq vector to letters model

I'm looking to build a sequence-to-sequence model that takes in a 2048-long vector of 1s and 0s as my input and translating it to my known output of (a variable length) 1-20 long characters (ex. ...