Skip to main content

Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0 votes
1 answer
50 views

How to improve classification accuracy in TF deep neural network model?

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow. I am trying to train a model on sequential data ...
0 votes
0 answers
5 views

RNN Formulation equivalence not clear

In the paper https://arxiv.org/pdf/1211.5063 the authors provide an alternative equation for the more widely known equation to calculate the hidden state at timestep $t$ $$ x_t = σ(W_{rec}x_{t−1} + W_{...
0 votes
1 answer
72 views

xLSTM parallel computation - mismatch in dimensions

In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
0 votes
0 answers
10 views

How to design a neural network where input consist of binary bits and output has discrete amplitudes?

The array size of input and output are different (e.g.: 88 - 176). Order of data points are matter it should not change. Also i want to make sure the reversibility like when i give amplitudes(...
0 votes
1 answer
33 views

Is sparsity required for behaviour exhibited by modular neural networks such as compositional generalisation, resiliency to catastrophic learning?

Having recurrency is intuitevely linked with modularity for me (you would need a network topology with an infinite number of layers to account for every combinaisons of situtations possible), but it's ...
0 votes
0 answers
6 views

An issue about the Decoder in seq2seq(rnn)

Here is a confusion about the decoder in seq2seq. In each time-step in decoder, there are two outputs: 1.output 2.hidden. and this hidden state is used as the next input hidden state. this output is ...
-1 votes
1 answer
57 views

Can anyone please explain the Recurrent Neural Network calculation shown in the picture?

As you can see, this is a recurrent neural network. I want to understand how the calculations are being made. Please, be as detailed as possible no matter how simple or self-explanatory the ...
0 votes
1 answer
42 views

Any reasons LSTM does not pick up any patterns?

I'm trying to teach an LSTM to predict the next values in 3 related series. (Financial data) Unfortunately, it looks like I made some basic mistake and this network never gets past just returning ...
4 votes
1 answer
232 views

Is there any relation between the recursive neural network and recurrent neural network?

Recurrent neural networks, abbreviated as RNNs, are widely used in deep learning literature, especially for text processing. Are they related to recursive neural networks in any way? I am asking for ...
1 vote
1 answer
63 views

Is image machine translation done in two steps?

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/...
0 votes
2 answers
127 views

Which NLP applications are based on recurrent neural networks?

Some of the NLP applications taken from this link NLP Applications: Machine Translation Speech Recognition Sentiment Analysis Question Answering Automatic Summarization Chatbots Market Intelligence ...
1 vote
1 answer
147 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
6 votes
1 answer
170 views

How to graphically represent a RNN architecture implemented in Keras?

I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say: ...
3 votes
2 answers
359 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
0 votes
1 answer
17 views

Best way to create a summary of variable length set of vectors where order does not matter

I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter. I'll have to ...
0 votes
1 answer
83 views

What underlying network is typically meant with ResNET?

When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks ...
0 votes
0 answers
57 views

Drum sound classification using RNN issues - help needed

I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound ...
0 votes
1 answer
129 views

How to produce documents like factset blackline?

Factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + ...
1 vote
2 answers
63 views

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at ...
1 vote
1 answer
46 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
1 vote
2 answers
893 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
0 votes
1 answer
414 views

What model can solve vector to vector prediction?

I am totally newbie into serial prediction. I am think about which model or AI paradigm can be used to do vector to vector prediction? For instance, [1,0,1] ^ [0,1,0] = [1,1,1] Another example could ...
0 votes
0 answers
26 views

"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book

I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote: Neural networks excel ...
0 votes
1 answer
24 views

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
0 votes
1 answer
69 views

How LSTM really decide what to forget and not?

Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" ...
1 vote
1 answer
48 views

What is the appropriate RNN structure to do Sentiment Analysis with multiple dependent ratings?

Suppose we are doing sentiment analysis for a restaurant. Customers can rate the restaurant by #1: how expensive the restaurant is, #2:how good is the food and #3: how likely they will come again. The ...
2 votes
1 answer
58 views

Can teacher forcing in RNN ensure Turing completeness?

RNN has the same capability as a universal Turing machine. But I am confused whether RNN holds the same capabilities if we use teacher forcing. Consider the following excerpts from paragraphs taken ...
3 votes
1 answer
497 views

What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper?

Researchers at Stanford University released, in 2012, the paper Financial Market Time Series Prediction with Recurrent Neural Networks. It goes on to discuss how they used echo state networks to ...
16 votes
4 answers
30k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
1 vote
1 answer
142 views

Why do we use a delay when feeding our input data to the echo state network?

I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have ...
1 vote
2 answers
78 views

Attention with Recurrent Neural Networks

In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each ...
0 votes
0 answers
11 views

Input encoding in RNNs

I'm working on developing a Recurrent Neural Network (RNN) that performs the following task: there are 4 lights, and during each trial, one of them is turned on, followed by another one. The goal is ...
0 votes
0 answers
18 views

Why use auto-regressive models for time-series?

This is a naive question... But I realized that auto-regressive predictions can be inherently unstable due to previous prediction error monotonically accumulating in the inputs: $M(h_{t-n},...,h_{t-m},...
0 votes
0 answers
21 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
0 votes
0 answers
22 views

Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?

I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?
6 votes
3 answers
3k views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
2 votes
3 answers
1k views

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If ...
0 votes
0 answers
10 views

What do RNN neural networks lack in nowcasting time series?

I want to write a master thesis on nowcasting GDP? Has this been used and if so I don't fully understand how the neural networks should be built if I forecast quarterly GDP and link that to ...
1 vote
1 answer
114 views

How does the linear layer step work and what should I do at the end of the LSTM? [closed]

So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
0 votes
0 answers
63 views

Are there guidelines or rules of thumb on how to stack hidden layers in a RNN?

I’m currently working on the prediction of chaotic data and I have decided to see how well would an RNN, namely an LSTM, would do. I am fairly new to the topic of Neural Networks, but I have found a ...
0 votes
0 answers
98 views

What is the meaning of "dimensionality of the embeddings and hidden states"?

I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
2 votes
1 answer
62 views

How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function The task to solve is the following: ...
0 votes
0 answers
14 views

How can a RNN with 256 cells accept a input of any size?

I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model. ...
0 votes
2 answers
194 views

Can an RNN predict a sinus curve with no input?

I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values. I am wondering if ...
5 votes
2 answers
1k views

LLM-like architecture capable of dynamically learning from its own output

Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
0 votes
0 answers
24 views

Seeking methods to incorporate arbitrary actuator faults for Control Optimization

I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
1 vote
1 answer
33 views

LSTM with multiple data streams

I am working on the following problem: I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
0 votes
0 answers
16 views

Feeding variable length of 2D image slices of the MRI into the deep neural network

I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
0 votes
0 answers
52 views

Is my 1D signal using CNN & RNN regression reasonable?

I want to know if my impact-echo signals are proper with CNN or RNN regression model. I got some simulated signal, as following shows. In previous research, people mostly consider frequency or even ...
0 votes
0 answers
36 views

Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?

Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution. In essence, the RNN is actually learning a ...

1
2 3 4 5
8