Questions tagged [gated-recurrent-unit]
For questions related to the gated recurrent unit (GRU), a modification and simplification of the LSTM unit, which is a more sophisticated unit (with respect to the standard one) of a recurrent neural network (RNN). An RNN that uses GRU units is often called a GRU network. GRUs were introduced in the paper "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation" (2014) by Kyunghyun Cho et al.
20
questions
0
votes
0
answers
22
views
"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book
I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote:
Neural networks excel ...
- 101
0
votes
0
answers
18
views
Why use auto-regressive models for time-series?
This is a naive question... But I realized that auto-regressive predictions can be inherently unstable due to previous prediction error monotonically accumulating in the inputs: $M(h_{t-n},...,h_{t-m},...
- 424
0
votes
0
answers
52
views
Feature extraction from log-mel spectrograms using CNNs
I am currently working on an ASR-related project in which I would like to combine Convolutional Neural Network with GRU Network and CTC loss function.
The idea is to use the CNN to extract ...
- 1
1
vote
0
answers
44
views
The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination
I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model:
This model first archived about 0.03 accuracy score and gradually ...
- 11
1
vote
1
answer
67
views
Which calculation to use for GRU
Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies.
When calculating a cell there are as many legitimate resources which ...
- 38
0
votes
0
answers
205
views
How to visualize the input and output of the GRU cell?
GRU belongs to the family of recurrent neural networks. This family of neural networks works on sequence data.
But, it is taking time for me to understand the differences between sequence length and ...
- 3,840
0
votes
1
answer
103
views
Why does validation accuracy stop rising so soon?
I have implemented a GRU to deal with youtube comment data. I am a bit confused about why the validation score seems to even out around 70% and then keeps rising, this doesn't look like overfitting ...
- 1
0
votes
1
answer
135
views
Is my intuition about RNN wrong?
Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs.
Consider a task where you need to predict 0 if the previous input was 1. For example: ...
- 109
2
votes
1
answer
896
views
Multiple GRU layers to improve a text generation
I am using the model in this colab https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/text_generation.ipynb#scrollTo=AM2Uma_-yVIq for Shakespeare like text generation.
...
- 121
0
votes
0
answers
150
views
What do RNN, LSTM, and GRU layers do in Tensorflow?
I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, ...
- 111
0
votes
1
answer
54
views
What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
- 569
2
votes
0
answers
104
views
Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?
When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN.
Lets say instead of using dense ...
- 2,890
1
vote
0
answers
2k
views
What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?
I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...
- 51
3
votes
0
answers
423
views
What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?
Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer.
I have a pretty basic idea how the complexity of ...
- 51
2
votes
0
answers
47
views
Incorporating domain knowledge into recurrent network
I am currently trying to solve a classification task with a recurrent artificial neural network (RNN).
Situation
There are up to 350 inputs (X) mapped on one categorical output (y)(13 differnt ...
- 31
6
votes
1
answer
2k
views
What's the difference between LSTM and GRU?
I have been reading about LSTMs and GRUs, which are recurrent neural networks (RNNs). The difference between the two is the number and specific type of gates that they have. The GRU has an update gate,...
- 1,263
1
vote
1
answer
41
views
Inner working of Bidirectional RNNs
I'm trying to understand how Bidirectional RNNs work.
Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
- 125
4
votes
0
answers
286
views
RNN models displays upper limit on predictions
I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
- 41
3
votes
1
answer
3k
views
How do I choose the size of the hidden state of a GRU?
I'm trying to understand how the size of the hidden state affects the GRU.
For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
- 1,128
6
votes
2
answers
393
views
Why are GRU and LSTM better than standard RNNs?
It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.
