Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

162 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
1 answer
166 views

How to graphically represent a RNN architecture implemented in Keras?

I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say: ...
Mindaugas Bernatavičius's user avatar
6 votes
0 answers
184 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
musako's user avatar
  • 181
4 votes
1 answer
229 views

Is there any relation between the recursive neural network and recurrent neural network?

Recurrent neural networks, abbreviated as RNNs, are widely used in deep learning literature, especially for text processing. Are they related to recursive neural networks in any way? I am asking for ...
hanugm's user avatar
  • 3,890
4 votes
0 answers
291 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
Kornephoros's user avatar
4 votes
0 answers
116 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
adamconkey's user avatar
4 votes
0 answers
203 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
AIguy122's user avatar
3 votes
0 answers
98 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
3 votes
0 answers
117 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
Pluviophile's user avatar
  • 1,273
3 votes
0 answers
52 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
ADSBJason's user avatar
3 votes
0 answers
41 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
Erick Medina's user avatar
3 votes
0 answers
26 views

How to back propagate for implementation of Sequence-to-Sequence with Multi Decoders

I am proposing a modified version of Sequence-to-Sequence model with dual decoders. The problem that I am trying to solve is Neural Machine Translation into two languages at once. This is the ...
dee cue's user avatar
  • 143
3 votes
0 answers
57 views

Ideas on a network that can translate image differences into motor commands?

I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
SumakuTension's user avatar
3 votes
0 answers
266 views

How does bidirectional encoding allow the predicted word to indirectly "see itself"?

Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
HLeb's user avatar
  • 579
3 votes
0 answers
126 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
Romain Egele's user avatar
3 votes
2 answers
359 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
LP0's user avatar
  • 83
3 votes
1 answer
496 views

What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper?

Researchers at Stanford University released, in 2012, the paper Financial Market Time Series Prediction with Recurrent Neural Networks. It goes on to discuss how they used echo state networks to ...
BleedObsidian's user avatar
2 votes
0 answers
298 views

Why is it said transformers are more parallelizable than RNN's?

The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
Erick Macias's user avatar
2 votes
0 answers
93 views

What are Reservoir computers used for today?

Reservoir computers were very popular in the early 2000s. From what I understand, the advantage of reservoir computers is that, as opposed to generic recurrent neural networks, training is only done ...
Nathaël's user avatar
2 votes
0 answers
373 views

Why use a fully connected layer for attention?

In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
Vishaal's user avatar
  • 121
2 votes
1 answer
58 views

Can teacher forcing in RNN ensure Turing completeness?

RNN has the same capability as a universal Turing machine. But I am confused whether RNN holds the same capabilities if we use teacher forcing. Consider the following excerpts from paragraphs taken ...
hanugm's user avatar
  • 3,890
2 votes
0 answers
50 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
Jake B.'s user avatar
  • 181
2 votes
0 answers
31 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
Minh Khôi's user avatar
2 votes
0 answers
115 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
chessprogrammer's user avatar
2 votes
0 answers
97 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
Joe Black's user avatar
  • 181
2 votes
0 answers
190 views

Can you use transformer models to do autocomplete tasks?

I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
user1234544's user avatar
2 votes
0 answers
21 views

How do CNNs or RNNs "stack the feature of nodes by a specific order"?

I am trying to understand the following statement taken from the paper Graph Neural Networks: A Review of Methods and Applications (2019). Standard neural networks like CNNs and RNNs cannot handle ...
Jacob B's user avatar
  • 247
2 votes
0 answers
57 views

Reservoir of LSM vs. FF-NN or ELM

The reservoir of the Liquid State Machine is an array of random numbers connected to each other with a probability depending on the distance between each other. Because of this connection with each ...
eenn's user avatar
  • 21
2 votes
0 answers
1k views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
user1234544's user avatar
2 votes
0 answers
248 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
mgb's user avatar
  • 31
2 votes
0 answers
586 views

How do I train a multiple-speaker model (speech synthesis) based on Tacotron 2 and espnet?

I'm new to Speech Synthesis & Deep Learning. Recently, I got a task as described below: I have problem in training a multi-speaker model which should be created by Tacotron2. And I was told I can ...
Envelo Lee's user avatar
2 votes
0 answers
43 views

How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?

This post continues the topic in the following post: Is it possible to train a neural network with 3 inputs and 12 outputs?. I conducted several experiments in MATLAB and selected those neural ...
dtn's user avatar
  • 239
2 votes
0 answers
32 views

How are batch statistics computed in Recurrent Batch Normalization?

I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, ...
OverLordGoldDragon's user avatar
2 votes
0 answers
284 views

What is a location-based addressing in a neural Turing machine?

In the neural Turing machine (NTM), the content-based addressing and location-based addressing is used for memory addressing. Content-based addressing is similar to the attention-based model, ...
eno's user avatar
  • 21
2 votes
0 answers
389 views

Why is the transformer for time series forecasting faster than RNN?

I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
razvanc92's user avatar
  • 1,138
2 votes
0 answers
143 views

Why validation performance is unstable for my LSTM based model (labelling problems)?

I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem. The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is ...
hellmean's user avatar
  • 140
2 votes
0 answers
58 views

How can my Neural Network categorize message strings?

Abstract I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. ...
Anilla's user avatar
  • 131
2 votes
0 answers
135 views

Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?

I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG. While reading ...
mayank agrawal's user avatar
2 votes
0 answers
112 views

Combine two embeddding inputs to increase more performance in LSTM model

The situation I encountered here is that I have two inputs(for instance, image embedding, etc.) into the first lstm of a series of lstms to predict the next word to generate sentence(from the second ...
user14947's user avatar
2 votes
0 answers
73 views

Detecting symmetry in small images with RNN

My network works on 32x32 normalized (translationally) but noisy images. Its task it to determine whether image has simple symmetry (horizontal/vertical). It needs to be reasonably robust to rotation (...
Andrew Butenko's user avatar
2 votes
0 answers
50 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
Sylvain Cloutier's user avatar
2 votes
0 answers
69 views

Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?

I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron ...
JustAGuy's user avatar
2 votes
0 answers
82 views

seq2seq vector to letters model

I'm looking to build a sequence-to-sequence model that takes in a 2048-long vector of 1s and 0s as my input and translating it to my known output of (a variable length) 1-20 long characters (ex. ...
rajkarthikkumar's user avatar
2 votes
0 answers
175 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...
Juliano Oliveira's user avatar
1 vote
0 answers
101 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 211
1 vote
1 answer
131 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
FELIPE_RIBAS's user avatar
1 vote
1 answer
46 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
Krusty the Clown's user avatar
1 vote
0 answers
63 views

About RNN followed by dense layer

In RNN we do get one output for each time stamp of input right i.e. if we give input as "I am Good" we get three outputs representing I followed by am and Good so if we connect a dense layer ...
Sowrabh meduri's user avatar
1 vote
0 answers
52 views

Surrogate model to produce time series from parameter set

Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
Mich55's user avatar
  • 11
1 vote
0 answers
130 views

How is it possible to use batches of data from within the same sequence with an LSTM?

ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless? Using the example of an LSTM, it has a hidden ...
Recessive's user avatar
  • 1,396
1 vote
0 answers
95 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
Recessive's user avatar
  • 1,396