Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
2
votes
0answers
29 views

How do I calculate the number of trainable parameters of an LSTM model?

I am currently learning about LSTMs, and I was working through this tutorial. I created a network with an LSTM layer just as described in the tutorial: ...
3
votes
1answer
342 views

Why does the error of my LSTM not decrease after 10 epochs?

Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution. ...
2
votes
1answer
49 views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
0
votes
1answer
30 views

ValueError: Error when checking input: expected time_distributed_6_input to have 5 dimensions, but got array with shape (32, 224, 224, 3) [closed]

I'm trying to apply data augmentation to avoid overffiting in my CNN-LSTM image classification model. My training data has the shape : (1882, 1, 224, 224, 3) My ...
3
votes
1answer
117 views

Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
51
votes
4answers
84k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
2
votes
0answers
76 views

Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?

Background I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the ...
1
vote
1answer
71 views

time-series prediction : loss going down, then stagnates with very high variance

I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in [0,13]. I have noticed that one-hot encoding it and ...
2
votes
1answer
88 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
4
votes
1answer
202 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-...
1
vote
1answer
145 views

Why won't my model train with CTC loss?

I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy ...
1
vote
1answer
2k views

What’s the difference between LSTM and RNN?

What's the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
5
votes
3answers
2k views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
0
votes
0answers
12 views

How sensitive are LSTM's to random zero values in its target feature when training?

I have worked with lstm's in the past, specifically for time series forecasting. However, the target feature in these time series were relatively "stable". With the loosely defined "...
0
votes
0answers
8 views

Categorical Location based Time Series data Prediction using LSTMs

I have some time series data with ActivityType as location shared below. Each CaseID has 6 unique values with different ...
5
votes
2answers
822 views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
5
votes
2answers
7k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
0
votes
1answer
243 views

Feeding YOLOv4 image data into LSTM layer?

How would one extract the feature vector from a given input image using YOLOv4 and pass that data into an LSTM to generate captions for the image? I am trying to make an image captioning software in ...
2
votes
1answer
266 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
0
votes
0answers
18 views

How Long Can BPTT Truncated?

I wanted to ask what is, in general, the maximum value (the order of magnitude) of the number of time steps I can back-propagate in the past using TBTT (Truncated Backpropagation Through Time) in an ...
1
vote
0answers
27 views

Is this a good implementation of this LSTM architecture?

I had been looking at some OCR problems and came across this presentation. I implemented it. In the presentation, there is the LSTM-Stack (diagram and algorithm, slide 32): Here is a visualization of ...
1
vote
1answer
754 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
3
votes
2answers
58 views

Problem extracting features from convolutional layer where the dimensions are big for feature maps

I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the ...
0
votes
0answers
16 views

Boundedness of cell states in MC-LSTM (mass-conserving LSTM)

I'm currently reading Hoedt et al's paper on mass-conserving LSTM. In the corollary it is stated that "[T]he memory cells, $c_k^\tau$, are bounded by the sum of mass inputs $\sum_{t=1}^\tau x^t+...
7
votes
1answer
762 views

Is LSTM a subcategory of RNN?

Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different? Literature doesn't seem to be unitary on this. This figure appears to explain the models to be alternatives, but I ...
1
vote
0answers
422 views

What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?

I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...
2
votes
0answers
27 views

Machine learning with raw data alone / or raw data with its statistics

My question is very general and it does not originate from a specific problem. Let's assume that, through experience, we have learned that some statistical property of a set of data is important in ...
0
votes
1answer
23 views

Do LSTM in tensorflow work sequentially or in parallel

I have a basic understanding how a cell and a layer of an LSTM works. However, I get confused by what "number of units" (as termed in tensorflow) exactly means. A unit is, as far as I ...
3
votes
1answer
29 views

What type of neural network do you need if you want to detect an action or dynamic pattern instead of a static pattern?

Let's say that you want to detect if a man is running, walking, or dancing instead of just detecting a man still. What type of neural networks will you use for this purpose?
0
votes
1answer
42 views

Reinforcing Learning when action has no effect on the environment

I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment. ...
1
vote
1answer
415 views

Is this LSTM model underfitting?

I think this model is underfitting. Is this correct? ...
1
vote
0answers
38 views

What's the best way to feed stories to a neural network?

I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because ...
0
votes
2answers
25 views

What is the reason for a training loss that drops but validation that NEVER does

I've been working on learning about NLP via a beginners competition on Kaggle. I first trained a model with an embedding layer and then a simple linear layer. I actually got way better than a flip of ...
0
votes
1answer
54 views

Using a Neural Network (LSTM) to approve/reject word-type sequences

I would like to train an LSTM neural network to either "approve" or "reject" a string based on the word-type sequence. For instance: "Mike's Airplane" would output "...
1
vote
0answers
14 views

The results are not correct when predicting the future for a very long period of time with LSTM

I am currently using LSTM to try to predict future data in AirPassengers.csv. This is current code op my Colab (sorry for the comments are Japanese) https://colab.research.google.com/drive/...
0
votes
0answers
17 views

Time series forecasting with some challenges

I'm attempting to devise a strategy to make time series forecasts based on costs accumulated over time. My dataset contains about 7500 time-series sequences (call it an instance for now), each having ...
13
votes
4answers
11k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
0
votes
0answers
35 views

LSTM and CNN - feature engineering and order for time series classification

My questions are related to multivariate time series classification, hence it may differ from forecasting problems. I can have either variable (entire history of the series) or fixed time steps (...
1
vote
1answer
33 views

What is the input to the left most LSTM cell c(t-1) and h(t-1)?

Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?
0
votes
0answers
12 views

LSTM predictions are one time step lagging

My problem involves electricity prediction (time-series problem) for 1-hour ahead. I am using LSTM to forecast. Length of Dataset: 1 year at one-hour interval Input: Outdoor Temperature (Ot), ...
1
vote
1answer
39 views

How to define a "don't care" class in time series classification in Pytorch?

This is a theoretical question. Setup I have a time series classification task in which I should output a classification of 3 classes for every time stamp t. All ...
2
votes
1answer
101 views

How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
0
votes
1answer
147 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
0answers
27 views

Time series forecasting for multiple objects with common features

I know the title of this question may raise an eyebrow, but I can't find the technical terms to define or investigate the actual problem. To demonstrate my problem with a simple hypothetical scenario: ...
0
votes
0answers
30 views

How Long Short Term Memory (LSTM) work for time series classification?

I first got the concept of LSTM on how it works word to word prediction etc. However, I want to know how it work with the time-series classification. For example I have the follwing data (see image ...
0
votes
0answers
18 views

How will actual labels be matched with predicted labels when LSTM discards data even from current time stamp input data?

I read the tutorial of LSTM from here. However, I have certain doubts that I need to address. Since we use true labels and do not remove anything from the original data, then how is it possible for ...
1
vote
1answer
49 views

Why doesn't the LSTM model improve the time-series forecasting significantly with respect to the MLP model?

I have recently started learning time series forecasting. I have a dataset of the weekly payment history of 10k clients over 1 year, and I want to predict the future 5 payments for a test set of 1k ...
0
votes
0answers
13 views

End-to-end learning using LSTM-AE

I want to use prediction models like LSTM-AE to predict time-series data. The feature that the neural network should learn is in frequency between 40-60Hz. So, in order to learn the feature more ...
1
vote
1answer
29 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...

1
2 3 4 5 6