Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0 votes
1 answer
56 views

How LSTM really decide what to forget and not?

Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" ...
0 votes
0 answers
22 views

Using Keras Tuner to tune a Stock Prediction Classifier [closed]

I'm trying to create a classifier using an LSTM that will generate a 1 if a stock is going to go up or a 0 if it won't. I'm using keras tuner to generate the hyperparameters on a dataset of almost 20,...
1 vote
1 answer
47 views

What is the appropriate RNN structure to do Sentiment Analysis with multiple dependent ratings?

Suppose we are doing sentiment analysis for a restaurant. Customers can rate the restaurant by #1: how expensive the restaurant is, #2:how good is the food and #3: how likely they will come again. The ...
2 votes
1 answer
57 views

Can teacher forcing in RNN ensure Turing completeness?

RNN has the same capability as a universal Turing machine. But I am confused whether RNN holds the same capabilities if we use teacher forcing. Consider the following excerpts from paragraphs taken ...
3 votes
1 answer
492 views

What is the significance of this Stanford University "Financial Market Time Series Prediction with RNN's" paper?

Researchers at Stanford University released, in 2012, the paper Financial Market Time Series Prediction with Recurrent Neural Networks. It goes on to discuss how they used echo state networks to ...
14 votes
4 answers
28k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
1 vote
1 answer
140 views

Why do we use a delay when feeding our input data to the echo state network?

I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have ...
0 votes
1 answer
44 views

How to improve classification accuracy in TF deep neural network model?

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow. I am trying to train a model on sequential data ...
1 vote
2 answers
69 views

Attention with Recurrent Neural Networks

In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each ...
0 votes
0 answers
9 views

Input encoding in RNNs

I'm working on developing a Recurrent Neural Network (RNN) that performs the following task: there are 4 lights, and during each trial, one of them is turned on, followed by another one. The goal is ...
0 votes
0 answers
17 views

Why use auto-regressive models for time-series?

This is a naive question... But I realized that auto-regressive predictions can be inherently unstable due to previous prediction error monotonically accumulating in the inputs: $M(h_{t-n},...,h_{t-m},...
0 votes
0 answers
16 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
0 votes
0 answers
9 views

Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?

I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?
6 votes
3 answers
3k views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
2 votes
3 answers
968 views

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If ...
4 votes
1 answer
208 views

Is there any relation between the recursive neural network and recurrent neural network?

Recurrent neural networks, abbreviated as RNNs, are widely used in deep learning literature, especially for text processing. Are they related to recursive neural networks in any way? I am asking for ...
1 vote
1 answer
61 views

Is image machine translation done in two steps?

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/...
0 votes
2 answers
119 views

Which NLP applications are based on recurrent neural networks?

Some of the NLP applications taken from this link NLP Applications: Machine Translation Speech Recognition Sentiment Analysis Question Answering Automatic Summarization Chatbots Market Intelligence ...
0 votes
0 answers
9 views

What do RNN neural networks lack in nowcasting time series?

I want to write a master thesis on nowcasting GDP? Has this been used and if so I don't fully understand how the neural networks should be built if I forecast quarterly GDP and link that to ...
1 vote
1 answer
81 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
6 votes
1 answer
155 views

How to graphically represent a RNN architecture implemented in Keras?

I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say: ...
3 votes
2 answers
353 views

How to build my own dataset and model for an LSTM neural network

I have a sort of mathematical problem and I'm not sure which model I should choose to make an LSTM neural network. Currently in my country, there is a system in which certain groups of researchers ...
0 votes
1 answer
71 views

What underlying network is typically meant with ResNET?

When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks ...
1 vote
1 answer
70 views

How does the linear layer step work and what should I do at the end of the LSTM? [closed]

So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
0 votes
0 answers
61 views

Are there guidelines or rules of thumb on how to stack hidden layers in a RNN?

I’m currently working on the prediction of chaotic data and I have decided to see how well would an RNN, namely an LSTM, would do. I am fairly new to the topic of Neural Networks, but I have found a ...
2 votes
0 answers
84 views

What is the meaning of "dimensionality of the embeddings and hidden states"?

I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
0 votes
1 answer
121 views

How to produce documents like factset blackline?

Factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + ...
1 vote
1 answer
40 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
1 vote
2 answers
820 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
2 votes
1 answer
61 views

How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function The task to solve is the following: ...
0 votes
1 answer
348 views

What model can solve vector to vector prediction?

I am totally newbie into serial prediction. I am think about which model or AI paradigm can be used to do vector to vector prediction? For instance, [1,0,1] ^ [0,1,0] = [1,1,1] Another example could ...
0 votes
1 answer
21 views

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
0 votes
0 answers
14 views

How can a RNN with 256 cells accept a input of any size?

I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model. ...
0 votes
2 answers
145 views

Can an RNN predict a sinus curve with no input?

I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values. I am wondering if ...
4 votes
2 answers
1k views

LLM-like architecture capable of dynamically learning from its own output

Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
0 votes
0 answers
24 views

Seeking methods to incorporate arbitrary actuator faults for Control Optimization

I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
1 vote
1 answer
30 views

LSTM with multiple data streams

I am working on the following problem: I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
0 votes
0 answers
13 views

Feeding variable length of 2D image slices of the MRI into the deep neural network

I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
0 votes
0 answers
34 views

Is my 1D signal using CNN & RNN regression reasonable?

I want to know if my impact-echo signals are proper with CNN or RNN regression model. I got some simulated signal, as following shows. In previous research, people mostly consider frequency or even ...
0 votes
0 answers
33 views

Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?

Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution. In essence, the RNN is actually learning a ...
0 votes
0 answers
15 views

RNN - Time To Stop extimation given non-categorical events sequence

I have a dataset which contains sequences of event of the following type: Timestamp_s Timestamp_e Event_type I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count]...
0 votes
1 answer
43 views

Temporally Non-Aware RNN

I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension. I'...
0 votes
1 answer
50 views

Neural network for recognizing ship types based on location series

I am building a neural network for recognizing ship types based on a 1000-long series of location data (latitude-longitude, normalized to account for different km/longitude° metrics, so that vector ...
0 votes
0 answers
16 views

Neural Network (NN) to predict the duration (in seconds) of a fault, how to recognize the elapsed time in prediction of an active fault?

I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges: Model Performance: My neural network seems to be ...
3 votes
1 answer
315 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
13 votes
4 answers
5k views

Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?

Why LLMs learn so fast during inference, but, ironically, are so slow during training? That is, if you teach an AI a new concept in a prompt, it will learn and use the concept perfectly and flawless, ...
0 votes
0 answers
20 views

How do I find a similar RNN as a starting point?

I am new to machine learning and neural networks and I want to create a neural network for a study project. I would like to create a RNN, that uses one (A) or several time series (with the same length,...
0 votes
0 answers
50 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
0 votes
0 answers
93 views

How to improve CRNN model(CTC loss) accuracy for OCR task?

I take this as baseline model. The main difference on RNN part, ...
1 vote
1 answer
193 views

Why does my "entropy generation" RNN do so badly?

I'm new to relatively RNNs, and I'm trying to train generative and guessing neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle&...

1
2 3 4 5
8