The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [recurrent-neural-networks]

Use for questions about Recurrent Neural Networks

Filter by
Sorted by
Tagged with
1
vote
3answers
65 views

How can neural networks be used to generate rather than classify?

In my experience with Neural Nets, I have only used them to take input vectors and return binary output. But, here in a video, https://youtu.be/ajGgd9Ld-Wc?t=214, Kai Fu Lee, renowned AI Expert shows ...
0
votes
1answer
59 views

Emotional Speech Synthesis

We are a team of computer science our graduation project about EmotionalSpeech Synthesis. We've found valuable information like research papers and WaveNet, Tacotron. A website (https://www.voicery....
1
vote
1answer
54 views

Recurrent Neural Network to track distance from origin

I have a game/simulation that takes a vector of encoded sequences of moves (up, down, left, right). Let's say that these are sequential step taken by an ant moving in a 2D space, starting from the ...
0
votes
0answers
7 views

Why word embedding such as word2vec is not used as the output layer of a seq2seq decoder?

It would make sense to make the decoder predict a smaller embedding vector instead of softmax over a large dictionary. The word having the most cosine similarity with the output embedding could be ...
4
votes
1answer
30 views

dimensions of hidden layer and cell state layer in LSTM

I was following some examples to get familiar with tensorflow LSTM related api, but noticed that all LSTM initialization functions require only num_units parameter which denotes number of hidden units ...
0
votes
0answers
17 views

Better versions of LSTM

I know recurrent neural networks for example LSTM is one of them. After LSTM different variants of LSTM have come for example GRU. I don't know about newer RNN like LSTM and GRU, so I decided to ask ...
0
votes
1answer
19 views

How to change this RNN text classification code to become text generation code?

I can do text classification with RNN, in which the last output of RNN (rnn_outputs[-1]) is used to matmul with output layer weight and plus bias. That is getting a word (class name) after the last T ...
0
votes
1answer
29 views

How to map X to Y for TensorFlow RNN training data

Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data: X = [[0,0],[0,1],[1,0],[1,1]]; Y = [[0], [1], [1], [0] ]; ...
2
votes
1answer
56 views

Issue at training simple RNN for word generation

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. This is done with ...
1
vote
2answers
84 views

What are some examples of LSTM architectures?

I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" ...
2
votes
2answers
35 views

Conferences for Human Activity Recognition

What are some conferences for publishing papers on Deep Learning for Human Activity recognition? Do any of the major conferences have specific tracks for Human Activity Recognition?
1
vote
0answers
23 views

Is there an standard algorithm for giving options from an RNN?

You can feed books to an RNN and it learn how to produce text. What I'm interested in is an algorithm that, given say 20 letters it suggest, say the best 10 options for the next 10 letters. So for ...
1
vote
0answers
20 views

Literature on Sequence Regresssion

I have some rated time-sequential data and I would like to test if an ANN can learn a correlation between my measurements and ratings. I suspect I could just try a CNN where 1 Dimension is time or an ...
1
vote
1answer
48 views

What is an identity recurrent neural network?

What is an identity recurrent neural network (IRNN)? What is the difference between an IRNN and RNN?
2
votes
1answer
32 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
1
vote
0answers
14 views

A gated neural network for internal thought?

I have an idea for an RNN which has no separate internal memory state only an output. But there is a gate in which tells the neural network whether the output will be acted out in the physical world ...
2
votes
1answer
40 views

How can I keep context in my chatbot

I have created a chatbot by Keras based on movie dialog. I used RNN more specifically GRU . My bot can reply well. But the problem is , it can't hold the context . As an example if I say ...
4
votes
2answers
90 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
1
vote
1answer
44 views

Is there a rigorous proof for finding Hopfield minima?

I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just let the network keep ...
3
votes
2answers
124 views

How do layers in an artificial neural network transform inputs to outputs?

To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, ...
0
votes
0answers
44 views

Why doesnt my lstm model for time series prediction improve after certain level of performance?

I created an lstm model which predicts multioutput sequeances. It takes variable length sequences as input. These sequences are padded with zero to obtain equal length. Note that the time series are ...
6
votes
3answers
71 views

In sequence-to-sequence, why is the output of the decoder used as its input?

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence: Why is the output, ...
0
votes
0answers
17 views

Dynamic frames processing with CNN LSTM combination or otherwise

I have a unique implementation where I have to process videos with dynamic frame rates (that is the number of frames is different for each video in a batch). I am stacking all the frames in a single ...
0
votes
1answer
60 views

How can I stabilise a recurrent neural network used for binary classification?

I’m looking for some help with my neural network. I’m working on a binary classification on a recurrent neural network that predicts stock movements (up and down) Let’s say I’m studying Eur/Usd, I’m ...
0
votes
1answer
15 views

How to handle proper names or variable names in word2vec?

The input in word2vec is known word (spellings), each tagged by its ID. But if you process real text, there can be not only dictionary words but also proper nouns like human names, trade marks, file ...
2
votes
1answer
26 views

Changes in flow detection neural network?

Do you have any advice, what architecture of neural network is the best for following task? Let input be some (complex function), the neural network gains a flow of its values, so I guess there will ...
1
vote
0answers
40 views

RNN weights when varying the input size

I have a time-varying input size vector for a RNN. However, I am facing some difficulties understanding how to deal with my network weights when the input changes. Say we have a set of natural ...
0
votes
0answers
68 views

How Seq2Seq with Bidirectional RNN works?

First of all the scope of the question is as follows - we have Sequence2Sequence architecture with: Decoder: Bidirectional LSTM Encoder: regular (single directional) LSTM What I know: When you ...
1
vote
1answer
44 views

Are there neural networks that accept graphs or trees as inputs?

As far I know, the RNN accepts a sequence as input and can produce as a sequence as output. Are there neural networks that accept graphs or trees as inputs, so that to represent the relationships ...
1
vote
1answer
107 views

Do you need to store prevous values of weights and layers on recurrent layer while BPTT?

The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like ...
0
votes
2answers
103 views

Time series RNN vs DNN

Understandably RNNs are very good at solving problems involving audio, video and text processing due to arbitrary input length of this sort of data. What I don't understand is why RNNs are also ...
1
vote
1answer
30 views

Train and Test Accuracy of GRU network not increasing after 2nd epoch

So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel. My problem is about the classification of 8 hour long timeseries ...
3
votes
1answer
206 views

A mathematical explanation of Attention Mechanism

I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in ...
12
votes
2answers
2k views

What is a Recurrent Neural Network?

Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular NNs?
1
vote
0answers
70 views

How does bidirectional encoding allow the predicted word to indirectly “see itself”?

Before the release of BERT we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s ...
0
votes
0answers
15 views

Using the cloud service to trasform a picture using a neural algorithm?

yesterday I tried to transform a picture in the artistic style using CNNs based on A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge using a recent Torch ...
1
vote
0answers
38 views

What is the feasible neural network structure that can learn to identify types of trajectory of moving dots?

I have multiple image sequences, each of which contains an animation of two moving dots. The trajectory of the dots in a sequence is always cyclic (not necessarily circular). There are two types of ...
1
vote
2answers
76 views

Which algorithm should I use to map an input sentence to an output sentence?

I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what ...
0
votes
0answers
11 views

How is the length of an input sequence related to the structure of an RNN?

My question is only with regards to the feedforward part of an RNN. I am following these steps. I am working on prediction of a time series. The time series is a toy model generated by me. It is ...
0
votes
0answers
31 views

Which neural network to use for mapping a vector of size m to a vector of size n, where n >> m?

I am trying to solve a mapping problem on a grid (100x100) where I have few points, say 10, where I know the values of a tensor $\boldsymbol{M}$. I have a scalar field, $v$, which is related to the ...
3
votes
0answers
32 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
1
vote
0answers
24 views

Why validation performance is unstable for my LSTM based model (labelling problems)?

I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem. The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is ...
1
vote
0answers
37 views

Predicting sine using LSTM: Small output range and delayed output?

I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise. I have tweaked the model, changing the output layer ...
2
votes
0answers
40 views

How can my Neural Network categorize message strings?

Abstract I wish to design a neural network that will categorize messages based on criteria I have predefined. It should feature the ability to be proactively trained as it continues its lifecycle. ...
0
votes
2answers
66 views

Do I need an encoder-decoder architecture to predict the next item of a sequence?

I am trying to understand how RNNs are used for sequence modelling. On a tutorial here, it mentions that if you want to translate say a sentence from English to French you can use an encoder-decoder ...
3
votes
0answers
94 views

What is the intuition behind the calculation of the similarity between encoder and decoder states?

Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y_1, y_2, \dots, y_J$ from $x_1, x_2, \dots x_I$: $$P(...
1
vote
1answer
136 views

Hidden state of the GRU

I'm having a hard time understanding how does the size of the hidden state affects GRU. For example in a concrete example lets say I want to lean a GRU to count. I'm gonna feed it fx 3 timestamps the ...
1
vote
1answer
566 views

BERT - What are the segment and position embeddings used in here?

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo. ELMo paper - https://arxiv.org/pdf/1802.05365.pdf BERT paper - https://arxiv....
0
votes
1answer
80 views

how to predict the stock of food items using ai

Actually, I want to make an AI model which tells the seller about the maintaining stock of food items as a parameter of time and eventually learns by itself with the customer buying data (problem - ...
1
vote
0answers
20 views

Under What Conditions Will a Hopfield Network Tend to Converge to Spurious States?

I'm relatively new to neural networks, and I've been trying to program my own Hopfield network. I got it to the point where it can reliably reproduce a single pattern from a completely scrambled ...