Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

148 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
2answers
587 views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
4
votes
0answers
82 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
4
votes
0answers
149 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
4
votes
0answers
83 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
4
votes
2answers
545 views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
3
votes
0answers
45 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
3
votes
0answers
37 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
3
votes
0answers
31 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
3
votes
0answers
37 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
3
votes
1answer
291 views

Why does the error of my LSTM not decrease after 10 epochs?

Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution. ...
3
votes
0answers
73 views

Why don't the neural networks inside LSTM cells contain hidden layers?

I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers. Why don't LSTM cells have more complex neural ...
3
votes
0answers
53 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
3
votes
0answers
87 views

Why do regression LSTMs learn high to low inputs significantly better than low to high?

The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361. I have ...
3
votes
0answers
91 views

What sort of Neural Network is best suited to predicting a future purchase?

I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4. Now I'm trying to do a NN that can make a prediction. ...
3
votes
0answers
188 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
2answers
53 views

Problem extracting features from convolutional layer where the dimensions are big for feature maps

I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the ...
3
votes
0answers
196 views

Generating time series for doing time-series forecasting with LSTM

I have a .db file with columns as described below. This data has been collected by a software which monitors the file usage in a filesystem or in other words generates metadata about all the files in ...
3
votes
0answers
184 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-...
2
votes
0answers
24 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
2
votes
0answers
38 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
2
votes
0answers
46 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
2
votes
0answers
22 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
2
votes
0answers
28 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
2
votes
1answer
61 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
2
votes
0answers
85 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
2
votes
0answers
52 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
2
votes
0answers
70 views

How do I make my LSTM model more sensitive to changes in the sequence?

I have a many to one LSTM model for multiclass classification. For reference, this is the architecture of the model ...
2
votes
1answer
93 views

LSTM model on different time scales

I am a newbie to machine learning. I have an LSTM model that predicts the next output n+1 time 1, params 1, output 1 time 2, params 2, output 2 time 3, params 3, output 3 . . time n, params n, , ...
2
votes
0answers
27 views

Did people analyze dynamics of very simple LSTMs?

I wonder if researchers tried to understand how LSTMs work by analyzing the dynamics of simple LSTM (e.g. with 2 units)? For example how the hidden state evolves depending on the properties of weight ...
2
votes
0answers
26 views

How are batch statistics computed in Recurrent Batch Normalization?

I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, ...
2
votes
1answer
81 views

How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
2
votes
0answers
42 views

How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
2
votes
0answers
26 views

How can I detect fast and slow motion in videos?

I'm trying to detect if a given video shot is fast or slow motion. Basically, I need to calculate a "video motion" score in a given video sequence, meaning how fast or slow motion the video is. For ...
2
votes
0answers
128 views

Spike detection in time series using Artificial Neural Networks

I'm quite new in ANNs. I intend to use ANNs for predicting spike points in time series right before they happen. I've already used LSTM for another scenario, and I know that they can be used in ...
2
votes
1answer
223 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
2
votes
0answers
169 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
2
votes
0answers
43 views

RNN: Different test results on balanced and unbalanced data

I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each ...
2
votes
0answers
39 views

Multi-field text input for LSTM

I'm using LSTM to categorize medium-sized pieces of text. Each item to be categorized has several free-form text fields, in addition to several categorical fields. What is the best approach to using ...
2
votes
0answers
41 views

How does a neural network output text box location data?

I'm interested in creating a convolutional neural network or LSTM to locate text in an image. I don't want to OCR the text yet, just find the text regions. Yes, I know Tesseract and other systems can ...
2
votes
0answers
55 views

Is there a mechanism in the human brain that works analog to LSTMs?

Is there a mechanism in the human brain that works analog to LSTMs? Is there a biological/neuroscientific interpretation of LSTMs and recurrent neural networks? How do long-term and short-term ...
2
votes
1answer
110 views

Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
2
votes
0answers
105 views

Structure of a multilayered LSTM neural network?

I implemented a LSTM neural network in Pytorch. It worked but I want to know if it worked the way I guessed how it worked. Say there's a 2-layer LSTM network with 10 units in each layer. The inputs ...
2
votes
0answers
32 views

Does it make sense to add word embeddings as additional features for LSTM model?

I have an LSTM model. This model takes as input tokens. Those tokens represent XML markups extracted from some XML files. My model is working fine. However, I want to optimize it by adding word ...
2
votes
0answers
176 views

Deep learning model (LSTM) with temporal and non temporal attributes

I'm working on a project to predict the usage of all the files in a filesystem in near future based on the metadata of the file system for past 6 months. I've got the following attributes about the ...
2
votes
0answers
102 views

Combine two embeddding inputs to increase more performance in LSTM model

The situation I encountered here is that I have two inputs(for instance, image embedding, etc.) into the first lstm of a series of lstms to predict the next word to generate sentence(from the second ...
2
votes
0answers
70 views

seq2seq vector to letters model

I'm looking to build a sequence-to-sequence model that takes in a 2048-long vector of 1s and 0s as my input and translating it to my known output of (a variable length) 1-20 long characters (ex. ...
1
vote
0answers
24 views

LSTM Recursive Forecast

I am confused about the way the LSTM networks work when forecasting with a horizon that is not finite, but I'm rather searching for a prediction in whatever time in future. In physical terms, I would ...
1
vote
0answers
10 views

LSTM Forecast Evolution

I have a confusion about the way the LSTM networks work when forecasting with an horizon that is not finite but I'm rather searching for a prediction in whatever time in future. In physical terms I ...
1
vote
0answers
31 views

Forecasting of spatio-temporal event data

I’m currently working on my dissertation which is centred around forecasting social conflict events. I’m using data from GDELT (Global Database of Events, Tone, and Language) to develop my forecasting ...
1
vote
0answers
16 views

Is my dataset unlearnable, or is my LSTM model not smart enough?

I have time-series data obtained from a video. The data is composed of bitrate and corresponding label pairs for each timestamp: The distribution over the first 30 seconds is as follows: I have ...