Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [recurrent-neural-networks]

Use for questions about Recurrent Neural Networks

0
votes
2answers
40 views

Time series RNN vs DNN

Understandably RNNs are very good at solving problems involving audio, video and text processing due to arbitrary input length of this sort of data. What I don't understand is why RNNs are also ...
0
votes
1answer
12 views

Train and Test Accuracy of GRU network not increasing after 2nd epoch

So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel. My problem is about the classification of 8 hour long timeseries ...
3
votes
1answer
37 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
11
votes
2answers
2k views

What is a Recurrent Neural Network?

Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular NNs?
1
vote
0answers
42 views

How does bidirectional encoding allow the predicted word to indirectly “see itself”?

Before the release of BERT we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s ...
0
votes
0answers
15 views

Using the cloud service to trasform a picture using a neural algorithm?

yesterday I tried to transform a picture in the artistic style using CNNs based on A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge using a recent Torch ...
1
vote
0answers
36 views

What is the feasible neural network structure that can learn to identify types of trajectory of moving dots?

I have multiple image sequences, each of which contains an animation of two moving dots. The trajectory of the dots in a sequence is always cyclic (not necessarily circular). There are two types of ...
1
vote
2answers
56 views

Which algorithm should I use to map an input sentence to an output sentence?

I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what ...
0
votes
0answers
11 views

How is the length of an input sequence related to the structure of an RNN?

My question is only with regards to the feedforward part of an RNN. I am following these steps. I am working on prediction of a time series. The time series is a toy model generated by me. It is ...
0
votes
0answers
28 views

Which neural network to use for mapping a vector of size m to a vector of size n, where n >> m?

I am trying to solve a mapping problem on a grid (100x100) where I have few points, say 10, where I know the values of a tensor $\boldsymbol{M}$. I have a scalar field, $v$, which is related to the ...
3
votes
0answers
24 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
0
votes
0answers
15 views

Why validation performance is unstable for my LSTM based model (labelling problems)?

I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem. The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is ...
1
vote
0answers
33 views

Predicting sine using LSTM: Small output range and delayed output?

I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise. I have tweaked the model, changing the output layer ...
2
votes
0answers
38 views

How can my Neural Network categorize message strings?

Abstract I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. ...
0
votes
2answers
46 views

Do I need an encoder-decoder architecture to predict the next item of a sequence?

I am trying to understand how RNNs are used for sequence modelling. On a tutorial here, it mentions that if you want to translate say a sentence from English to French you can use an encoder-decoder ...
3
votes
0answers
84 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
1
vote
1answer
53 views

Hidden state of the GRU

I'm having a hard time understanding how does the size of the hidden state affects GRU. For example in a concrete example lets say I want to lean a GRU to count. I'm gonna feed it fx 3 timestamps the ...
0
votes
1answer
242 views

BERT - What are the segment and position embeddings used in here?

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo. ELMo paper - https://arxiv.org/pdf/1802.05365.pdf BERT paper - https://arxiv....
0
votes
1answer
68 views

how to predict the stock of food items using ai

Actually, I want to make an AI model which tells the seller about the maintaining stock of food items as a parameter of time and eventually learns by itself with the customer buying data (problem - ...
1
vote
0answers
19 views

Under What Conditions Will a Hopfield Network Tend to Converge to Spurious States?

I'm relatively new to neural networks, and I've been trying to program my own Hopfield network. I got it to the point where it can reliably reproduce a single pattern from a completely scrambled ...
2
votes
1answer
105 views

Additive Attention in Convolutional Networks

Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures ...
1
vote
2answers
147 views

How to use RNN With Attention Mechanism on Non Textual Data?

Recurrent Neural Networks (RNN) With Attention Mechanism is generally used for Machine Translation and Natural Language Processing. In Python, implementation of RNN With Attention Mechanism is ...
0
votes
1answer
49 views

LSTM language model not working

I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons ...
-3
votes
1answer
280 views

RNN LSTM not converging with Adam

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one ...
1
vote
0answers
138 views

Number of nodes in hidden layer for LSTM

I am starting to learn LSTM by understanding how it is used for creating a char-RNN and had a fundamental question. Does the number of nodes in the hidden layer need to be the same as that of the ...
1
vote
1answer
104 views

Active Learning and RNN

We have a series of data which want to label part of each series. As we do not have any training data, we try using Active Learning as a solution. But, the problem is our classifier is something like ...
3
votes
1answer
50 views

Does an advanced Dialogue state tracking eliminate the need of intent classifier and slot filling models in dialogue systems/ chatbots?

I am learning to create a dialogue system. The various parts of such a system are Intent classifier, slot filling, Dialogue state tracking (DST), dialogue policy optimization and NLG. While reading ...
1
vote
0answers
57 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
1
vote
0answers
32 views

Train a recurrent neural network by concatenating time series. Is it safe?

As the title says, I want to train a Jordan network (i.e. a particular kind of recurrent neural network) using a certain number of time series. Let's say that $x_1, x_2, \ldots x_N$ are $N$ input ...
2
votes
0answers
91 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
-1
votes
1answer
24 views

Getting better results in improving the configuration

Currently, I found the right recipe for a time series regression problem to finally get acceptable to good results. Here is the config file ...
1
vote
2answers
163 views

What should I do when I have a variable-length sequence when instantiating an LSTM in Keras?

In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that ...
3
votes
1answer
296 views

Fourier Transform inputs (Frequency) for RNN

Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.
3
votes
1answer
766 views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
1
vote
1answer
55 views

predict waste generation

I am starting a project to predict the generation of urban waste. I have found very little information on this topic on the internet. I would be very useful advice on how to approach this topic, and ...
1
vote
1answer
65 views

Why would giving my AI more data make it perform worse?

So I trained an AI to generate shakespeare, which it did somewhat well. I used this 10,000 character sample. Next I tried to get it to generate limericks using these 100,000 limericks. It generated ...
2
votes
1answer
57 views

Is it possible to use an RNN to predict a feature that is not an input feature?

I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now. Let's say I have a set of input features, generated every second. Corresponding with ...
3
votes
2answers
208 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. And I specifically mean language model in its standard sense. I ...
1
vote
0answers
39 views

How recurrent neural network work when predict many days?

I use recurrent neural network, RNNs have to get input one value per step and it will show one value output. If I have daily sale demand time series data. I want to predict sale demand for three ...
4
votes
1answer
404 views

Will attention based networks prevail over RNN and LSTM?

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
1
vote
1answer
242 views

Backward Pass for LSTMs

TL;DR I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far. On pg. 21 of the ...
9
votes
4answers
475 views

Beyond neural networks?

Are there possible algorithms that have the potential to replace neural nets in the near future? And do we need that? What is the worst thing of using neural networks in terms of efficiency?
5
votes
2answers
595 views

Structure of LSTM RNNs

I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you! For ...
1
vote
0answers
119 views

Deep learning model (LSTM) with temporal and non temporal attributes

I'm working on a project to predict the usage of all the files in a filesystem in near future based on the metadata of the file system for past 6 months. I've got the following attributes about the ...
3
votes
2answers
122 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
1
vote
0answers
30 views

How to adapt RNNs to variable frequency / framerate of inputs?

Say I have an application where the frequency of the input is known but can vary widely across sequences. For example, they may be audio recordings acquired at different frequency, or videos that come ...
2
votes
1answer
35 views

Why are GRU and LSTM better than old types of RNN?

Seems older RNNs have a limitation for their use cases and have been outperformed by other architectures for specific tasks e.g GRUS and CNNs
3
votes
2answers
170 views

Is there an alternative to RNNs that doesn't require knowing input history?

In an RNN to train it, you need to roll it out, and enter in the history of inputs and the history of expected outcomes. This doesn't seem like a realistic picture of the brain since this would ...
-1
votes
1answer
44 views

Teaching a NN to manipulate pseudoRNG over a long time scale?

For speedrunning purposes, I am trying to train a neural network to identify human-executable ways to manipulate pseudo-RNG (in Pokemon Red, for the interested). The game runs at sixty frames per ...
4
votes
1answer
282 views

Over- and underestimations of the lowest and highest values in LSTM network

I'm training a LSTM network with multiple inputs and several LSTM layers in order to setup a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation on the ...