Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

What would be the total number of learnable parameters of the RNN encoder of this encoder-decoder architecture for machine translation?

Here's a quiz. My answer is different from the teacher's, so I'm wondering what answer would you pick up. We use a sequence-to-sequence (encoder-decoder) system to perform machine translation. We ...
5
votes
4answers
3k views

How can I predict the next number in a non-obvious sequence?

I've got an array of integers ranging from -3 to +3. Example: [1, 3, -2, 0, 0, 1] The array has no obvious pattern since it represents bipolar disorder mood swings. What is the most suitable approach ...
0
votes
0answers
20 views

Multiple GRU layers to improve a text generation

I am using the model in this colab https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/text_generation.ipynb#scrollTo=AM2Uma_-yVIq for Shakespeare like text generation. ...
1
vote
0answers
29 views
0
votes
0answers
5 views

Do the output of RNN individual layers go through Softmax when going from one layer to the next in a stacked RNN (many to one architecture)?

In most of the online materials that I've read, the equations of RNNs are shown only for a single layer RNN with the output going through softmax (for a many-to-one architecture). I am trying to find ...
0
votes
0answers
20 views

NLP problem Phrase/Token labeling

Looking for suggestions on how to define the following NLP problem and different ways in which it can be modeled to leverage machine learning. I believe there are multiple ways to model this problem. ...
0
votes
0answers
17 views

RNN - Backpropagation through time - Gradient Calculation

I think I got it right after reading multiple resources but im still not 100%. Seems like everyone is calculating it different. Or they just shortcut explaining the calculation. (or my math skills ...
7
votes
1answer
758 views

Is LSTM a subcategory of RNN?

Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different? Literature doesn't seem to be unitary on this. This figure appears to explain the models to be alternatives, but I ...
0
votes
0answers
16 views

Is my understanding of RNNs wrong?

I asked a similar question a few days back here, but since no one replied, I thought I should subdivide my question further. My understanding of RNNs is as follows, Suppose I have a standard MLP. To ...
0
votes
0answers
67 views

What's wrong with my understanding of how RNNs work?

Recently, I've been trying to derive the mathematics behind various Neural Network structures. I managed to derive the MLP and tested it to be on par with a Keras implementation (Using the MNIST ...
1
vote
1answer
28 views

How is Google Translate able to translate texts of arbitrarily large length?

Sequence-to-sequence models with attention are known to be limited by a maximum sequence length. So how can we handle sequences of arbitrarily large size? Do we just set a very large maximum sequence ...
0
votes
0answers
10 views

LSTM predictions are one time step lagging

My problem involves electricity prediction (time-series problem) for 1-hour ahead. I am using LSTM to forecast. Length of Dataset: 1 year at one-hour interval Input: Outdoor Temperature (Ot), ...
1
vote
1answer
20 views

Is reconciling shape discrepancies the only purpose of padding?

Padding is a technique used in some of the domains of artificial intelligence. Data is generally available in different shapes. But in order to pass the data as input to a model in deep learning, the ...
0
votes
0answers
13 views

Feeding the output back to input in 3D CNN model

I am currently designing a Model which takes Input 3D Grid and Model Output at $t-1$. The model figure is described below I have two thoughts in training the model for above situation. Feed output $...
0
votes
0answers
13 views

What is the significance of the RegLoss colum in Neuralprophet

I recently made a forecast with neuralprophet and after training, I got a table with three columns; "SmoothL1Loss", "MAE" and "RegLoss". Please, I need to know the ...
0
votes
2answers
63 views

How to design a neural network with arbitrary input and output length?

I am trying to build a neural network that has an input of $n$ pairs of integer values (where $n$ is random) and a corresponding output of a binary array with length $n$. The input will be a set of ...
1
vote
0answers
23 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
0
votes
1answer
10 views

Can we modelize an RNN by an ANN that takes precedent output as a part of input?

Is it possible to consider an RNN as a classical feedforward neural network that just take the precedent output as a part of the input ?
0
votes
0answers
16 views

Do bi-directional RNNs necessarily use 100% teacher forcing?

I typically think of teacher forcing as optional when training an RNN. We may either: use the output of time-step $t$ as the input to time-step $t+1$ use the $(t+1)$th input as the input to time-...
0
votes
0answers
16 views

Seq2Seq Models not used for NLP - input to the Decoder?

I am looking into Seq2Seq models but using it to make multi-step predictions of factory data and I am getting a little confused with the inputs to the Decoder model. Correct me if I am wrong, but the ...
0
votes
0answers
19 views

Good metrics and losses to use for Sequence-to-Sequence model for time-series prediction/forecasting

I am developing a sequence-to-sequence LSTM model for multi-step time series forecasting. I have the basic model working, so now I need to drill down on which loss function and evaluation metrics to ...
0
votes
0answers
42 views

What do RNN, LSTM, and GRU layers do in Tensorflow?

I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, ...
0
votes
0answers
23 views

Document clustering from ordered pages list

I have a series of ordered pdf pages which own to different documents. Let me give you an example: Pages: 1 2 3 4 5 6 True Pages: 1 2 | 1 2 3 4 So I have like six ordered pages, two of which from ...
0
votes
1answer
37 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
0
votes
1answer
25 views

Initial Input $h_0$ for RNN and updation of weights

Consider an input to RNN $ x = \{x_i\}_{1}^{n}$. Assume that the length of each input $x_i$ is k. Now, consider the following diagram from p5 of this pdf My doubts are: What should I pass as $h_0$? ...
0
votes
1answer
74 views

Is it possible to overfit a model on infinite amounts of data?

This is a theoretical question. Is it possible to overfit a model on infinite amounts of data? Let me clarify there are no duplicates. Say, we have a generator function that produces data, with the ...
1
vote
1answer
55 views

Rescaling time-series data with very spiky pattern for training data in LSTM network

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below)...
1
vote
0answers
29 views

Construction of GRUcell using Tensorflow to get the hidden states [closed]

I want to construct a GRU cell with one hidden layer but I want to get the hidden states at each time step. I want to train the GRU cell for let's say 10 times and at every step to get the hidden ...
0
votes
0answers
53 views

Is it sensible to combine GRU/LSTM with the transformer's encoder?

Is it sensible to combine GRU/LSTM with the transformer's encoder? If we take the output of a GRU (uni or bi-directional), and then feed it as input to the transformer's encoder, would that help in ...
0
votes
0answers
28 views

How to detect dynamic hand gestures?

I already know how to detect static hand gestures like fist, peace etc. I wonder however, how to detect dynamic hand gestures like swipe left/right or "draw" circle with hand. Is some kind ...
1
vote
0answers
16 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
0
votes
1answer
138 views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
0
votes
0answers
20 views

How to Classify Game Stages Based on Bitrate Time Series Data Using RNN - LSTM

I need suggestions for my project and would be glad if you would give me a hand. I have a dataset of frames obtained from the old-school game DOOM. Each frame in the dataset has the following columns: ...
1
vote
1answer
26 views

How to train an LSTM to classify based on rare historic event?

I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far. My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle ...
1
vote
1answer
42 views

Predict time series from initial non-time dependant parameters

I'm trying to create an algorithm (neural network) that is able to predict a time series from a set of different parameters that are not given through time. Let's say I have a plane flying under the ...
0
votes
0answers
27 views

Are there any inverse RNN layers?

Given the model: Sequence([ GRU(200, input_shape=(None,100), return_sequences=False) ]) Which maps the space ...
1
vote
0answers
31 views

Is a true RNN auto encoder possible with Keras/TF

I want to get some encodings for temporal data (with a highly varying number of timesteps). The dataset is of the format: ...
0
votes
1answer
51 views

Must all CNNs and RNNs not have a fully connected layer in order to be considered as such?

In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers ...
1
vote
2answers
81 views

What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?

I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. ...
1
vote
1answer
37 views

Do RNNs/LSTMs really need to be sequential?

There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
1
vote
1answer
47 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
-1
votes
1answer
27 views

Extracting values from text based on keywords

I am trying to read a PDF file and put it in Python string and trying to fetch information based on keywords. The text here is completely irregular. Example of text Blockquote Ram has taken an ...
0
votes
0answers
34 views

Multi dimensional LSTM modeling in KERAS

I have a database of time series signals with multiple features and Im trying to build a model to predict whether or not two samples are related to each other. For example : a database of 1000 sample ...
0
votes
0answers
23 views

Can RNNs be used to classify these time series into two classes?

My task is to classify into two classes the time series like these shown in the figure. The figure shows one class on the left sub-figure and second one on the right. The series are shown in pairs ...
0
votes
2answers
87 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
0
votes
0answers
22 views

Gradient of CTC Loss?

I am having a hard time figuring out how the gradient of the CTC loss function looks like. Could anyone explain that to me?
3
votes
0answers
50 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
2
votes
0answers
24 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
1
vote
1answer
101 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
0
votes
2answers
84 views

Can the hidden state of an RNN be a matrix?

If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?

1
2 3 4 5 6