Questions tagged [recurrent-neural-networks]
For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.
367
questions
2
votes
1
answer
61
views
Distinguishing between the fundamental structures of the convolutional neural network and the recurrent neural network: hierarchical vs sequential
I'm trying to distinguish between the fundamental structures of the convolutional neural network and the recurrent neural network. Convolutional neural networks build a hierarchical model from the ...
- 603
0
votes
0
answers
23
views
How does information traverse through a neural network such as LSTMs or ESNs?
In "The “echo state” approach to analysing and training recurrent neural networks-with an erratum note" (2001), H. Jaeger defines the "Echo State Network" (ESN). I have read that ...
- 101
0
votes
1
answer
31
views
Input size for a RNN layer by modeling a dynamical system
I'm trying to understand the concept of dynamical system with NN by studying this very well explanation here.
But I stuck in the following problem:
The author uses a RNN and/or a LSTM network as a NN ...
- 204
0
votes
0
answers
18
views
Covering missing derivations in Bengio RNN exploding gradient/output paper
While I have no doubts about the soundness of its conclusions given that it is very well-known, I have been looking through the classic paper on RNNs which is used to substantiate the claims that ...
- 101
1
vote
1
answer
46
views
What is the exact purpose of input modulation gate in LSTMs?
Basically, I was learning about LSTMs where I found LSTMs are made up of three gates: The forget gate, input gate and output gate. However, I came across some sources that state there is a fourth gate ...
- 11
1
vote
0
answers
14
views
RNN Formulation equivalence not clear
In the paper https://arxiv.org/pdf/1211.5063 the authors provide an alternative equation for the more widely known equation to calculate the hidden state at timestep $t$
$$
x_t = σ(W_{rec}x_{t−1} + W_{...
- 121
0
votes
0
answers
22
views
How to design a neural network where input consist of binary bits and output has discrete amplitudes?
The array size of input and output are different (e.g.: 88 - 176). Order of data points are matter it should not change. Also i want to make sure the reversibility like when i give amplitudes(...
0
votes
0
answers
8
views
An issue about the Decoder in seq2seq(rnn)
Here is a confusion about the decoder in seq2seq.
In each time-step in decoder, there are two outputs: 1.output 2.hidden.
and this hidden state is used as the next input hidden state.
this output is ...
- 1
0
votes
1
answer
44
views
Is sparsity required for behaviour exhibited by modular neural networks such as compositional generalisation, resiliency to catastrophic learning?
Having recurrency is intuitevely linked with modularity for me (you would need a network topology with an infinite number of layers to account for every combinaisons of situtations possible), but it's ...
- 101
-1
votes
1
answer
67
views
Can anyone please explain the Recurrent Neural Network calculation shown in the picture?
As you can see, this is a recurrent neural network. I want to understand how the calculations are being made. Please, be as detailed as possible no matter how simple or self-explanatory the ...
0
votes
0
answers
74
views
Any reasons LSTM does not pick up any patterns?
I'm trying to teach an LSTM to predict the next values in 3 related series. (Financial data)
Unfortunately, it looks like I made some basic mistake and this network never gets past just returning ...
- 111
0
votes
1
answer
108
views
xLSTM parallel computation - mismatch in dimensions
In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
- 101
0
votes
0
answers
67
views
Drum sound classification using RNN issues - help needed
I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound ...
- 1
1
vote
2
answers
125
views
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at ...
0
votes
1
answer
24
views
Best way to create a summary of variable length set of vectors where order does not matter
I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter.
I'll have to ...
- 101
0
votes
0
answers
32
views
"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book
I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote:
Neural networks excel ...
- 101
0
votes
1
answer
95
views
How LSTM really decide what to forget and not?
Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" ...
- 3
0
votes
0
answers
13
views
Input encoding in RNNs
I'm working on developing a Recurrent Neural Network (RNN) that performs the following task: there are 4 lights, and during each trial, one of them is turned on, followed by another one. The goal is ...
- 151
1
vote
2
answers
87
views
Attention with Recurrent Neural Networks
In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each ...
0
votes
0
answers
23
views
Why use auto-regressive models for time-series?
This is a naive question... But I realized that auto-regressive predictions can be inherently unstable due to previous prediction error monotonically accumulating in the inputs: $M(h_{t-n},...,h_{t-m},...
- 434
0
votes
0
answers
21
views
Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?
here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
0
votes
0
answers
22
views
Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?
I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?
- 11
2
votes
3
answers
1k
views
Why is the vanishing gradient problem especially relevant for a RNN and not a MLP
I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If ...
- 256
0
votes
0
answers
12
views
What do RNN neural networks lack in nowcasting time series?
I want to write a master thesis on nowcasting GDP? Has this been used and if so I don't fully understand how the neural networks should be built if I forecast quarterly GDP and link that to ...
- 11
0
votes
0
answers
109
views
What is the meaning of "dimensionality of the embeddings and hidden states"?
I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
user78686
2
votes
1
answer
64
views
How can I deal with random weights initialisation when predicting a time-series sine function?
I am training a simple RNN model in keras to predict a time series.
The time series I am considering is just a sine function
The task to solve is the following:
...
- 123
0
votes
0
answers
17
views
How can a RNN with 256 cells accept a input of any size?
I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model.
...
0
votes
0
answers
24
views
Seeking methods to incorporate arbitrary actuator faults for Control Optimization
I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
1
vote
1
answer
48
views
LSTM with multiple data streams
I am working on the following problem:
I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
- 11
0
votes
0
answers
18
views
Feeding variable length of 2D image slices of the MRI into the deep neural network
I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
- 101
0
votes
0
answers
62
views
Is my 1D signal using CNN & RNN regression reasonable?
I want to know if my impact-echo signals are proper with CNN or RNN regression model.
I got some simulated signal, as following shows.
In previous research, people mostly consider frequency or even ...
- 1
0
votes
0
answers
38
views
Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?
Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution.
In essence, the RNN is actually learning a ...
0
votes
0
answers
18
views
RNN - Time To Stop extimation given non-categorical events sequence
I have a dataset which contains sequences of event of the following type:
Timestamp_s
Timestamp_e
Event_type
I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count]...
- 31
0
votes
1
answer
43
views
Temporally Non-Aware RNN
I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension.
I'...
user55176
0
votes
0
answers
21
views
How do I find a similar RNN as a starting point?
I am new to machine learning and neural networks and I want to create a neural network for a study project.
I would like to create a RNN, that uses one (A) or several time series (with the same length,...
- 1
1
vote
1
answer
140
views
How does the linear layer step work and what should I do at the end of the LSTM? [closed]
So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
- 111
0
votes
0
answers
113
views
LSTM Ensemble: Combine low, mid, and high frequency time series data
I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
- 121
2
votes
0
answers
354
views
Why is it said transformers are more parallelizable than RNN's?
The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
- 21
0
votes
1
answer
26
views
Handcraft RNN with attention to extract central element
I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
- 101
0
votes
1
answer
2k
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would ...
- 103
0
votes
0
answers
31
views
How to use deep relative trust to measure the distance between two RNNs, and between two transformers?
I want to measure how different two networks, i.e., between two RNNs and between
two transformers. I read that Deep relative trust can be used to measure the distance between two NNs. Can it be used ...
- 165
0
votes
1
answer
41
views
RNNs - Is the recurrence at the layer-level or at the network-level?
I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.
...
- 103
3
votes
1
answer
103
views
In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?
I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each ...
0
votes
2
answers
219
views
Can an RNN predict a sinus curve with no input?
I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values.
I am wondering if ...
1
vote
0
answers
107
views
In terms of explainability, is attentive RNN easier to explain than the transformer?
Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
- 221
13
votes
4
answers
6k
views
Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?
Why LLMs learn so fast during inference, but, ironically, are so slow during training? That is, if you teach an AI a new concept in a prompt, it will learn and use the concept perfectly and flawless, ...
- 395
5
votes
2
answers
1k
views
LLM-like architecture capable of dynamically learning from its own output
Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
- 395
1
vote
1
answer
236
views
Recognize patterns within random sequences
I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
- 111
0
votes
1
answer
2k
views
Difference between sequence length and hidden size in LSTM
It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs.
For example, in the next ...
- 31
0
votes
1
answer
83
views
Seq2seq with RNNs, how is the training loop performed?
How do we train a seq2seq rnn training?
We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...