Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
3
votes
2answers
150 views

How do layers in an artificial neural network transform inputs to outputs?

To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, ...
1
vote
0answers
65 views

Why doesnt my lstm model for time series prediction improve after certain level of performance?

I created an lstm model which predicts multioutput sequeances. It takes variable length sequences as input. These sequences are padded with zero to obtain equal length. Note that the time series are ...
6
votes
3answers
83 views

In sequence-to-sequence, why is the output of the decoder used as its input?

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence: Why is the output, ...
1
vote
0answers
37 views

Dynamic frames processing with CNN LSTM combination or otherwise

I have a unique implementation where I have to process videos with dynamic frame rates (that is the number of frames is different for each video in a batch). I am stacking all the frames in a single ...
1
vote
1answer
74 views

How can I stabilise a recurrent neural network used for binary classification?

I’m looking for some help with my neural network. I’m working on a binary classification on a recurrent neural network that predicts stock movements (up and down) Let’s say I’m studying Eur/Usd, I’m ...
1
vote
1answer
24 views

How to handle proper names or variable names in word2vec?

The input in word2vec is known word (spellings), each tagged by its ID. But if you process real text, there can be not only dictionary words but also proper nouns like human names, trade marks, file ...
2
votes
1answer
29 views

Changes in flow detection neural network?

Do you have any advice, what architecture of neural network is the best for following task? Let input be some (complex function), the neural network gains a flow of its values, so I guess there will ...
1
vote
0answers
46 views

RNN weights when varying the input size

I have a time-varying input size vector for a RNN. However, I am facing some difficulties understanding how to deal with my network weights when the input changes. Say we have a set of natural ...
0
votes
0answers
179 views

How Seq2Seq with Bidirectional RNN works?

First of all the scope of the question is as follows - we have Sequence2Sequence architecture with: Decoder: Bidirectional LSTM Encoder: regular (single directional) LSTM What I know: When you ...
3
votes
2answers
90 views

Are there neural networks that accept graphs or trees as inputs?

As far I know, the RNN accepts a sequence as input and can produce as a sequence as output. Are there neural networks that accept graphs or trees as inputs, so that to represent the relationships ...
1
vote
1answer
133 views

Do you need to store prevous values of weights and layers on recurrent layer while BPTT?

The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like ...
1
vote
2answers
277 views

Time series RNN vs DNN

Understandably RNNs are very good at solving problems involving audio, video and text processing due to arbitrary input length of this sort of data. What I don't understand is why RNNs are also ...
1
vote
1answer
125 views

Train and Test Accuracy of GRU network not increasing after 2nd epoch

So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel. My problem is about the classification of 8 hour long timeseries ...
5
votes
1answer
543 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
12
votes
2answers
2k views

What is a recurrent neural network?

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular (or ...
3
votes
0answers
125 views

How does bidirectional encoding allow the predicted word to indirectly “see itself”?

Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
1
vote
0answers
16 views

Using the cloud service to trasform a picture using a neural algorithm?

yesterday I tried to transform a picture in the artistic style using CNNs based on A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge using a recent Torch ...
1
vote
0answers
40 views

What is the feasible neural network structure that can learn to identify types of trajectory of moving dots?

I have multiple image sequences, each of which contains an animation of two moving dots. The trajectory of the dots in a sequence is always cyclic (not necessarily circular). There are two types of ...
1
vote
2answers
142 views

Which algorithm should I use to map an input sentence to an output sentence?

I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what ...
4
votes
0answers
54 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
1
vote
0answers
45 views

Why validation performance is unstable for my LSTM based model (labelling problems)?

I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem. The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is ...
1
vote
0answers
40 views

Predicting sine using LSTM: Small output range and delayed output?

I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise. I have tweaked the model, changing the output layer ...
2
votes
0answers
45 views

How can my Neural Network categorize message strings?

Abstract I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. ...
0
votes
2answers
89 views

Do I need an encoder-decoder architecture to predict the next item of a sequence?

I am trying to understand how RNNs are used for sequence modelling. On a tutorial here, it mentions that if you want to translate say a sentence from English to French you can use an encoder-decoder ...
3
votes
0answers
104 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
2
votes
1answer
292 views

How do I choose the size of the hidden state of a GRU?

I'm trying to understand how the size of the hidden state affects the GRU. For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
1
vote
1answer
97 views

how to predict the stock of food items using ai

Actually, I want to make an AI model which tells the seller about the maintaining stock of food items as a parameter of time and eventually learns by itself with the customer buying data (problem - ...
1
vote
0answers
22 views

Under What Conditions Will a Hopfield Network Tend to Converge to Spurious States?

I'm relatively new to neural networks, and I've been trying to program my own Hopfield network. I got it to the point where it can reliably reproduce a single pattern from a completely scrambled ...
3
votes
1answer
339 views

Additive Attention in Convolutional Networks

Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures ...
1
vote
2answers
320 views

How to use RNN With Attention Mechanism on Non Textual Data?

Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is ...
1
vote
1answer
303 views

LSTM language model not working

I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons ...
-2
votes
1answer
1k views

RNN LSTM not converging with Adam

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one ...
1
vote
0answers
336 views

Number of nodes in hidden layer for LSTM

I am starting to learn LSTM by understanding how it is used for creating a char-RNN and had a fundamental question. Does the number of nodes in the hidden layer need to be the same as that of the ...
2
votes
1answer
265 views

How can active learning be used in the case of complex models that require a lot of data?

We have a series of data and we want to label the parts of each series. As we do not have any training data, we could try to use active learning as a solution, but the problem is that our classifier ...
3
votes
1answer
96 views

Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?

I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG. While reading ...
3
votes
0answers
107 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
1
vote
0answers
38 views

Train a recurrent neural network by concatenating time series. Is it safe?

As the title says, I want to train a Jordan network (i.e. a particular kind of recurrent neural network) using a certain number of time series. Let's say that $x_1, x_2, \ldots x_N$ are $N$ input ...
2
votes
0answers
100 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
-1
votes
1answer
24 views

Getting better results in improving the configuration

Currently, I found the right recipe for a time series regression problem to finally get acceptable to good results. Here is the config file ...
2
votes
2answers
409 views

What should I do when I have a variable-length sequence when instantiating an LSTM in Keras?

In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that ...
3
votes
1answer
724 views

Fourier Transform inputs (Frequency) for RNN

Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.
7
votes
2answers
4k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
1
vote
1answer
64 views

predict waste generation

I am starting a project to predict the generation of urban waste. I have found very little information on this topic on the internet. I would be very useful advice on how to approach this topic, and ...
1
vote
1answer
67 views

Why would giving my AI more data make it perform worse?

So I trained an AI to generate shakespeare, which it did somewhat well. I used this 10,000 character sample. Next I tried to get it to generate limericks using these 100,000 limericks. It generated ...
3
votes
1answer
122 views

Is it possible to use an RNN to predict a feature that is not an input feature?

I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now. Let's say I have a set of input features, generated every second. Corresponding with ...
1
vote
0answers
45 views

How recurrent neural network work when predict many days?

I use recurrent neural network, RNNs have to get input one value per step and it will show one value output. If I have daily sale demand time series data. I want to predict sale demand for three ...
6
votes
1answer
1k views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
2
votes
1answer
549 views

How to change the backward pass for an LSTM layer that outputs to another LSTM layer?

I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far. On pg. 21 of the pdf (pg. ...
10
votes
4answers
787 views

What are the models that have the potential to replace neural networks in the near future?

Are there possible models that have the potential to replace neural networks in the near future? And do we even need that? What is the worst thing about using neural networks in terms of efficiency?
7
votes
2answers
2k views

Structure of LSTM RNNs

I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you! For ...