Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Feeding the output back to input in 3D CNN model

I am currently designing a Model which takes Input 3D Grid and Model Output at $t-1$. The model figure is described below I have two thoughts in training the model for above situation. Feed output $...
0
votes
0answers
15 views

Pandas dataframe to keras LSTM input shape [closed]

I have a simple dataset consisting of only two columns (year and price of oil). Now, I need to shape them in order for keras' LSTM-layer to accept their input_shape. My code looks like this, I ...
0
votes
0answers
7 views

What is the significance of the RegLoss colum in Neuralprophet

I recently made a forecast with neuralprophet and after training, I got a table with three columns; "SmoothL1Loss", "MAE" and "RegLoss". Please, I need to know the ...
0
votes
2answers
46 views

How to design a neural network with arbitrary input and output length?

I am trying to build a neural network that has an input of $n$ pairs of integer values (where $n$ is random) and a corresponding output of a binary array with length $n$. The input will be a set of ...
1
vote
0answers
22 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
0
votes
1answer
10 views

Can we modelize an RNN by an ANN that takes precedent output as a part of input?

Is it possible to consider an RNN as a classical feedforward neural network that just take the precedent output as a part of the input ?
0
votes
0answers
9 views

Do bi-directional RNNs necessarily use 100% teacher forcing?

I typically think of teacher forcing as optional when training an RNN. We may either: use the output of time-step $t$ as the input to time-step $t+1$ use the $(t+1)$th input as the input to time-...
0
votes
0answers
13 views

Seq2Seq Models not used for NLP - input to the Decoder?

I am looking into Seq2Seq models but using it to make multi-step predictions of factory data and I am getting a little confused with the inputs to the Decoder model. Correct me if I am wrong, but the ...
0
votes
0answers
17 views

Good metrics and losses to use for Sequence-to-Sequence model for time-series prediction/forecasting

I am developing a sequence-to-sequence LSTM model for multi-step time series forecasting. I have the basic model working, so now I need to drill down on which loss function and evaluation metrics to ...
0
votes
0answers
36 views

What do RNN, LSTM, and GRU layers do in Tensorflow?

I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, ...
0
votes
0answers
23 views

Document clustering from ordered pages list

I have a series of ordered pdf pages which own to different documents. Let me give you an example: Pages: 1 2 3 4 5 6 True Pages: 1 2 | 1 2 3 4 So I have like six ordered pages, two of which from ...
0
votes
1answer
35 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
0
votes
1answer
25 views

Initial Input $h_0$ for RNN and updation of weights

Consider an input to RNN $ x = \{x_i\}_{1}^{n}$. Assume that the length of each input $x_i$ is k. Now, consider the following diagram from p5 of this pdf My doubts are: What should I pass as $h_0$? ...
0
votes
1answer
63 views

Is it possible to overfit a model on infinite amounts of data?

This is a theoretical question. Is it possible to overfit a model on infinite amounts of data? Let me clarify there are no duplicates. Say, we have a generator function that produces data, with the ...
1
vote
1answer
37 views

Rescaling time-series data with very spiky pattern for training data in LSTM network

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below)...
1
vote
0answers
28 views

Construction of GRUcell using Tensorflow to get the hidden states [closed]

I want to construct a GRU cell with one hidden layer but I want to get the hidden states at each time step. I want to train the GRU cell for let's say 10 times and at every step to get the hidden ...
0
votes
0answers
43 views

Is it sensible to combine GRU/LSTM with the transformer's encoder?

Is it sensible to combine GRU/LSTM with the transformer's encoder? If we take the output of a GRU (uni or bi-directional), and then feed it as input to the transformer's encoder, would that help in ...
0
votes
0answers
28 views

How to detect dynamic hand gestures?

I already know how to detect static hand gestures like fist, peace etc. I wonder however, how to detect dynamic hand gestures like swipe left/right or "draw" circle with hand. Is some kind ...
1
vote
0answers
12 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
0
votes
1answer
91 views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
0
votes
0answers
20 views

How to Classify Game Stages Based on Bitrate Time Series Data Using RNN - LSTM

I need suggestions for my project and would be glad if you would give me a hand. I have a dataset of frames obtained from the old-school game DOOM. Each frame in the dataset has the following columns: ...
1
vote
1answer
25 views

How to train an LSTM to classify based on rare historic event?

I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far. My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle ...
1
vote
1answer
34 views

Predict time series from initial non-time dependant parameters

I'm trying to create an algorithm (neural network) that is able to predict a time series from a set of different parameters that are not given through time. Let's say I have a plane flying under the ...
0
votes
0answers
24 views

Are there any inverse RNN layers?

Given the model: Sequence([ GRU(200, input_shape=(None,100), return_sequences=False) ]) Which maps the space ...
1
vote
0answers
28 views

Is a true RNN auto encoder possible with Keras/TF

I want to get some encodings for temporal data (with a highly varying number of timesteps). The dataset is of the format: ...
0
votes
1answer
49 views

Must all CNNs and RNNs not have a fully connected layer in order to be considered as such?

In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers ...
1
vote
2answers
78 views

What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?

I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. ...
1
vote
1answer
37 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
1
vote
1answer
47 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
-1
votes
1answer
24 views

Extracting values from text based on keywords

I am trying to read a PDF file and put it in Python string and trying to fetch information based on keywords. The text here is completely irregular. Example of text Blockquote Ram has taken an ...
0
votes
0answers
19 views

Multi dimensional LSTM modeling in KERAS

I have a database of time series signals with multiple features and Im trying to build a model to predict whether or not two samples are related to each other. For example : a database of 1000 sample ...
0
votes
0answers
23 views

Can RNNs be used to classify these time series into two classes?

My task is to classify into two classes the time series like these shown in the figure. The figure shows one class on the left sub-figure and second one on the right. The series are shown in pairs ...
0
votes
2answers
86 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
0
votes
0answers
19 views

Gradient of CTC Loss?

I am having a hard time figuring out how the gradient of the CTC loss function looks like. Could anyone explain that to me?
2
votes
0answers
42 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
2
votes
0answers
24 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
1
vote
1answer
58 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
0
votes
2answers
63 views

Can the hidden state of an RNN be a matrix?

If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?
0
votes
0answers
13 views

How does teacher forcing with output feedback works in the context of Echo State Networks and signal generation?

I understood the idea behind Echo State Networks. Given a signal, for each timestep the amplitude is fed into a recurrent network and the activation of each unit is measured. The activation vector is ...
0
votes
1answer
25 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised ...
0
votes
1answer
80 views

Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?

There were some posts that using RNN can predict the next point of the sine wave function with data history. However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
0
votes
0answers
37 views

Seq2Seq Modelling: when implementing some machine translation net, how are special tokens embedded?

When implementing any encoder-decoder network for machine translation, during training we provide the true output sentence to the decoder so that the context vector (from source language) may be ...
0
votes
1answer
51 views

What error should I use for RNN?

I'm relatively new to machine learning, and I don't know what error I should use for an RNN. I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next ...
4
votes
1answer
266 views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
0
votes
1answer
46 views

Factors that causing totally different outcomes from an exactly same model and datasets

Here is a model that trains time series data in (batch, step, features) way. I have kept the random state for train test split function the same. Every parameter below the same, running the model ...
4
votes
2answers
524 views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
0
votes
1answer
63 views

Is the working of RNNs, LSTM and GRU sequential or parallel?

You take any blog or any example and all they tell you about is the given picture below. It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in ...
2
votes
0answers
23 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
0
votes
0answers
23 views

Is there a reference that describes Recurrent Neural Networks for NLP tasks?

I would like some references of works that try to understand the functioning of any kind of RNN in natural language processing tasks. They can be any work that tries to explain the functioning of the ...
0
votes
0answers
63 views

Training speed in GPU vs CPU for LSTM

I was experimenting seq2seq model which was the bi-LSTM encoder/decoder with attention. When I compared the training times on GPU vs CPU while varying the batch size, I got CPU on the small batch ...

1
2 3 4 5 6