Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
-1 votes
1 answer
47 views

Can anyone please explain the Recurrent Neural Network calculation shown in the picture?

As you can see, this is a recurrent neural network. I want to understand how the calculations are being made. Please, be as detailed as possible no matter how simple or self-explanatory the ...
Syed_Hamza_Akbar_Ali's user avatar
0 votes
1 answer
37 views

Any reasons LSTM does not pick up any patterns?

I'm trying to teach an LSTM to predict the next values in 3 related series. (Financial data) Unfortunately, it looks like I made some basic mistake and this network never gets past just returning ...
viraptor's user avatar
  • 111
0 votes
1 answer
46 views

xLSTM parallel computation - mismatch in dimensions

In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
Quaere Verum's user avatar
0 votes
0 answers
53 views

Drum sound classification using RNN issues - help needed

I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound ...
APasagic's user avatar
1 vote
2 answers
56 views

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at ...
Vladislav Korecký's user avatar
0 votes
1 answer
16 views

Best way to create a summary of variable length set of vectors where order does not matter

I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter. I'll have to ...
hasdrubal's user avatar
  • 101
0 votes
0 answers
25 views

"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book

I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote: Neural networks excel ...
Kaira's user avatar
  • 101
0 votes
1 answer
65 views

How LSTM really decide what to forget and not?

Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" ...
Ashraf's user avatar
  • 3
0 votes
0 answers
11 views

Input encoding in RNNs

I'm working on developing a Recurrent Neural Network (RNN) that performs the following task: there are 4 lights, and during each trial, one of them is turned on, followed by another one. The goal is ...
Blade's user avatar
  • 151
1 vote
2 answers
78 views

Attention with Recurrent Neural Networks

In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each ...
FluidMechanics Potential Flows's user avatar
0 votes
0 answers
18 views

Why use auto-regressive models for time-series?

This is a naive question... But I realized that auto-regressive predictions can be inherently unstable due to previous prediction error monotonically accumulating in the inputs: $M(h_{t-n},...,h_{t-m},...
profPlum's user avatar
  • 424
0 votes
0 answers
20 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
Jeremy Kenn's user avatar
0 votes
0 answers
19 views

Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?

I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?
J_Bake's user avatar
  • 11
2 votes
3 answers
1k views

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If ...
PeterBe's user avatar
  • 256
0 votes
0 answers
10 views

What do RNN neural networks lack in nowcasting time series?

I want to write a master thesis on nowcasting GDP? Has this been used and if so I don't fully understand how the neural networks should be built if I forecast quarterly GDP and link that to ...
J_Bake's user avatar
  • 11
0 votes
0 answers
96 views

What is the meaning of "dimensionality of the embeddings and hidden states"?

I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
user avatar
0 votes
0 answers
63 views

Are there guidelines or rules of thumb on how to stack hidden layers in a RNN?

I’m currently working on the prediction of chaotic data and I have decided to see how well would an RNN, namely an LSTM, would do. I am fairly new to the topic of Neural Networks, but I have found a ...
Jxson99's user avatar
  • 103
2 votes
1 answer
62 views

How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function The task to solve is the following: ...
apt45's user avatar
  • 123
0 votes
0 answers
14 views

How can a RNN with 256 cells accept a input of any size?

I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model. ...
D Star Let's Explore's user avatar
0 votes
0 answers
24 views

Seeking methods to incorporate arbitrary actuator faults for Control Optimization

I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
IsolatedSushi's user avatar
1 vote
1 answer
32 views

LSTM with multiple data streams

I am working on the following problem: I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
PenguinHook's user avatar
0 votes
0 answers
16 views

Feeding variable length of 2D image slices of the MRI into the deep neural network

I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
Oleg's user avatar
  • 101
0 votes
0 answers
49 views

Is my 1D signal using CNN & RNN regression reasonable?

I want to know if my impact-echo signals are proper with CNN or RNN regression model. I got some simulated signal, as following shows. In previous research, people mostly consider frequency or even ...
hui30319's user avatar
0 votes
0 answers
34 views

Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?

Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution. In essence, the RNN is actually learning a ...
Akash Tiwari's user avatar
0 votes
0 answers
18 views

RNN - Time To Stop extimation given non-categorical events sequence

I have a dataset which contains sequences of event of the following type: Timestamp_s Timestamp_e Event_type I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count]...
GPU'njoyer's user avatar
0 votes
1 answer
43 views

Temporally Non-Aware RNN

I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension. I'...
user avatar
0 votes
0 answers
19 views

Neural Network (NN) to predict the duration (in seconds) of a fault, how to recognize the elapsed time in prediction of an active fault?

I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges: Model Performance: My neural network seems to be ...
Tools's user avatar
  • 1
0 votes
0 answers
20 views

How do I find a similar RNN as a starting point?

I am new to machine learning and neural networks and I want to create a neural network for a study project. I would like to create a RNN, that uses one (A) or several time series (with the same length,...
xlaub's user avatar
  • 1
0 votes
0 answers
52 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
llllvvuu's user avatar
  • 146
1 vote
1 answer
101 views

How does the linear layer step work and what should I do at the end of the LSTM? [closed]

So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
Anish Kommireddy's user avatar
0 votes
0 answers
108 views

How to improve CRNN model(CTC loss) accuracy for OCR task?

I take this as baseline model. The main difference on RNN part, ...
4daJKong's user avatar
  • 101
0 votes
0 answers
64 views

LSTM Ensemble: Combine low, mid, and high frequency time series data

I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
Ai4l2s's user avatar
  • 121
2 votes
0 answers
298 views

Why is it said transformers are more parallelizable than RNN's?

The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
Erick Macias's user avatar
0 votes
1 answer
24 views

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
Daraan's user avatar
  • 101
0 votes
1 answer
2k views

Which preprocessing is the correct way to forecast time-series data using LSTM?

I just started to study time-series forecasting using RNN. I have a few months of time series data that was an hour unit. The data is a kind of percentage value of my little experiment and I would ...
orde.r's user avatar
  • 103
0 votes
0 answers
12 views

Neural translation model - alignment as a latent variable or as a trainable part of the model

I consider the neural machine translation method in https://arxiv.org/pdf/1409.0473.pdf. I understand that the intuition here, as the title says, is to jointly train alignment and translation. ...
kiriloff's user avatar
  • 121
0 votes
0 answers
27 views

How to use deep relative trust to measure the distance between two RNNs, and between two transformers?

I want to measure how different two networks, i.e., between two RNNs and between two transformers. I read that Deep relative trust can be used to measure the distance between two NNs. Can it be used ...
Sanyou's user avatar
  • 165
0 votes
1 answer
39 views

RNNs - Is the recurrence at the layer-level or at the network-level?

I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch. ...
BPDev's user avatar
  • 103
0 votes
0 answers
37 views

Is this lstm diagram correct?

I made an LSTM diagram, but do not know if it is correct. Can you point out any errors in case there are any?
David H. J.'s user avatar
3 votes
1 answer
71 views

In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?

I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each ...
Mike Arsenault's user avatar
0 votes
2 answers
183 views

Can an RNN predict a sinus curve with no input?

I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values. I am wondering if ...
jollytall's user avatar
1 vote
0 answers
101 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 211
13 votes
4 answers
5k views

Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?

Why LLMs learn so fast during inference, but, ironically, are so slow during training? That is, if you teach an AI a new concept in a prompt, it will learn and use the concept perfectly and flawless, ...
MaiaVictor's user avatar
5 votes
2 answers
1k views

LLM-like architecture capable of dynamically learning from its own output

Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
MaiaVictor's user avatar
1 vote
1 answer
130 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
FELIPE_RIBAS's user avatar
0 votes
1 answer
1k views

Difference between sequence length and hidden size in LSTM

It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs. For example, in the next ...
moth123's user avatar
  • 31
0 votes
1 answer
77 views

Seq2seq with RNNs, how is the training loop performed?

How do we train a seq2seq rnn training? We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
FluidMechanics Potential Flows's user avatar
1 vote
1 answer
46 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
Krusty the Clown's user avatar
0 votes
1 answer
45 views

How to improve classification accuracy in TF deep neural network model?

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow. I am trying to train a model on sequential data ...
Fr_nkenstien's user avatar
2 votes
1 answer
157 views

Why do training and fixing a reservoir yield very similar results (in an echo state network)?

Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered. I am trying to better understand how echo state networks work. To see, how fixing the weights of the ...
user avatar

1
2 3 4 5
8