Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
1
vote
0answers
19 views

Construction of GRUcell using Tensorflow to get the hidden states

I want to construct a GRU cell with one hidden layer but I want to get the hidden states at each time step. I want to train the GRU cell for let's say 10 times and at every step to get the hidden ...
0
votes
0answers
28 views

Is it sensible to combine GRU/LSTM with the transformer's encoder?

Is it sensible to combine GRU/LSTM with the transformer's encoder? If we take the output of a GRU (uni or bi-directional), and then feed it as input to the transformer's encoder, would that help in ...
0
votes
0answers
28 views

How to detect dynamic hand gestures?

I already know how to detect static hand gestures like fist, peace etc. I wonder however, how to detect dynamic hand gestures like swipe left/right or "draw" circle with hand. Is some kind ...
1
vote
0answers
10 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
0
votes
1answer
44 views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
0
votes
0answers
20 views

How to Classify Game Stages Based on Bitrate Time Series Data Using RNN - LSTM

I need suggestions for my project and would be glad if you would give me a hand. I have a dataset of frames obtained from the old-school game DOOM. Each frame in the dataset has the following columns: ...
1
vote
1answer
25 views

How to train an LSTM to classify based on rare historic event?

I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far. My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle ...
1
vote
1answer
31 views

Predict time series from initial non-time dependant parameters

I'm trying to create an algorithm (neural network) that is able to predict a time series from a set of different parameters that are not given through time. Let's say I have a plane flying under the ...
0
votes
0answers
23 views

Are there any inverse RNN layers?

Given the model: Sequence([ GRU(200, input_shape=(None,100), return_sequences=False) ]) Which maps the space ...
1
vote
0answers
28 views

Is a true RNN auto encoder possible with Keras/TF

I want to get some encodings for temporal data (with a highly varying number of timesteps). The dataset is of the format: ...
0
votes
1answer
46 views

Must all CNNs and RNNs not have a fully connected layer in order to be considered as such?

In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers ...
1
vote
2answers
78 views

What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?

I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. ...
1
vote
1answer
31 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
1
vote
1answer
42 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
-1
votes
1answer
23 views

Extracting values from text based on keywords

I am trying to read a PDF file and put it in Python string and trying to fetch information based on keywords. The text here is completely irregular. Example of text Blockquote Ram has taken an ...
0
votes
0answers
17 views

Multi dimensional LSTM modeling in KERAS

I have a database of time series signals with multiple features and Im trying to build a model to predict whether or not two samples are related to each other. For example : a database of 1000 sample ...
0
votes
0answers
22 views

Can RNNs be used to classify these time series into two classes?

My task is to classify into two classes the time series like these shown in the figure. The figure shows one class on the left sub-figure and second one on the right. The series are shown in pairs ...
0
votes
2answers
82 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
0
votes
0answers
15 views

Gradient of CTC Loss?

I am having a hard time figuring out how the gradient of the CTC loss function looks like. Could anyone explain that to me?
2
votes
0answers
34 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
2
votes
0answers
23 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
1
vote
1answer
42 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
0
votes
2answers
44 views

Can the hidden state of an RNN be a matrix?

If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?
0
votes
0answers
13 views

How does teacher forcing with output feedback works in the context of Echo State Networks and signal generation?

I understood the idea behind Echo State Networks. Given a signal, for each timestep the amplitude is fed into a recurrent network and the activation of each unit is measured. The activation vector is ...
0
votes
1answer
25 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised ...
0
votes
1answer
80 views

Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?

There were some posts that using RNN can predict the next point of the sine wave function with data history. However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
0
votes
0answers
37 views

Seq2Seq Modelling: when implementing some machine translation net, how are special tokens embedded?

When implementing any encoder-decoder network for machine translation, during training we provide the true output sentence to the decoder so that the context vector (from source language) may be ...
0
votes
1answer
51 views

What error should I use for RNN?

I'm relatively new to machine learning, and I don't know what error I should use for an RNN. I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next ...
4
votes
1answer
219 views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
0
votes
1answer
46 views

Factors that causing totally different outcomes from an exactly same model and datasets

Here is a model that trains time series data in (batch, step, features) way. I have kept the random state for train test split function the same. Every parameter below the same, running the model ...
4
votes
2answers
175 views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
0
votes
1answer
47 views

Is the working of RNNs, LSTM and GRU sequential or parallel?

You take any blog or any example and all they tell you about is the given picture below. It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in ...
2
votes
0answers
23 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
0
votes
0answers
23 views

Is there a reference that describes Recurrent Neural Networks for NLP tasks?

I would like some references of works that try to understand the functioning of any kind of RNN in natural language processing tasks. They can be any work that tries to explain the functioning of the ...
0
votes
0answers
49 views

Training speed in GPU vs CPU for LSTM

I was experimenting seq2seq model which was the bi-LSTM encoder/decoder with attention. When I compared the training times on GPU vs CPU while varying the batch size, I got CPU on the small batch ...
0
votes
0answers
99 views

How are temporal links made between following sequences in RNN?

Say I use an RNN, whatever is the cell's type, to perform time series classification. It can thus be seen as sequence classification. The time series is split into random, equal size, overlapping ...
0
votes
0answers
9 views

Is the information in the hiddenstate of a RNN worth processing further after the input passes the RNN?

I hope the question is understandable. I just wanted to ask if the hidden state, which is passed through the timesteps/cells of an RNN/LSTM/GRU to deliver information from $\text{cell}_{i-1}$ to the $\...
0
votes
0answers
22 views

What is $\frac{du}{dt}$ represent in recurrent neural network

I am currently studying Recurrent neural network for solving optimization problems, which is popular back in the 1990s. I am very familiar with the traditional feedforward neural network, however, ...
0
votes
0answers
78 views

Understanding Stable Baselines Custom Policies

I'm trying to understand the structure of the custom recurrent policy introduced in the documentation of the Stable Baselines: How exactly is the Lstm NN constructed? (check code below) From what I ...
1
vote
0answers
24 views

How do I infer exploding or vanishing gradients in Keras?

It may already be obvious that I am just a practitioner and just a beginner to Deep Learning. I am still figuring out lots of "WHY"s and "HOW"s of DL. So, for example, if I train a ...
2
votes
1answer
36 views

Understanding LSTM through example

I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states. Lets assume that ...
4
votes
1answer
41 views

How to graphically represent a RNN architecture implemented in Keras?

I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say: ...
2
votes
0answers
37 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
1
vote
0answers
45 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
1
vote
1answer
289 views

How is Google Translate able to convert texts of different lengths?

According to my experience with Tensorflow and many other frameworks, neural networks have to have a fixed shape for any output, but how does Google translate convert texts of different lengths?
1
vote
2answers
50 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
3
votes
1answer
60 views

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information? What ...
1
vote
1answer
42 views

How does vanish gradient restrict RNN to not work for long range dependencies?

I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like: Consider ...
1
vote
1answer
23 views

How do LSTMs work if the following two matrices are not able to be multiplied?

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance: The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$ The input data at timestep t ...
1
vote
0answers
166 views

What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?

I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...

1
2 3 4 5 6