Questions tagged [long-short-term-memory]
For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".
126
questions with no upvoted or accepted answers
6
votes
0
answers
154
views
Why Pixel RNN (Row LSTM) can capture triangular contexts?
I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts?
In this paper,
the kernel of the one-dimensional convolution ...
- 181
6
votes
2
answers
2k
views
What evaluation metric are used for sequence-to-sequence prediction problems?
I am solving many sequence-to-sequence prediction problems using RNN/LSTM.
What type of evaluation metrics can be used for sequence prediction problems?
One metric is the mean squared error (MSE) ...
- 181
4
votes
0
answers
116
views
Could zero-padding affect learning in a negative way?
I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
- 143
4
votes
0
answers
245
views
RNN models displays upper limit on predictions
I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
- 41
4
votes
0
answers
109
views
How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?
It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
- 271
4
votes
4
answers
942
views
Use Machine/Deep Learning to Guess a String
I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc). ...
- 49
3
votes
0
answers
75
views
How does backpropagation work in LSTMs?
After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
- 1,223
3
votes
0
answers
44
views
Is there any way of generating fixed-length sequences with RNNs?
Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
- 51
3
votes
0
answers
41
views
How to predict an event (or action) based on a window of time-series measurements?
I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static.
I ...
- 31
3
votes
0
answers
90
views
Why don't the neural networks inside LSTM cells contain hidden layers?
I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers.
Why don't LSTM cells have more complex neural ...
- 31
3
votes
0
answers
135
views
How can I do hyperparameter optimization for a CNN-LSTM neural network?
I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
- 31
3
votes
0
answers
88
views
Why do regression LSTMs learn high to low inputs significantly better than low to high?
The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361.
I have ...
- 1,326
3
votes
0
answers
99
views
What sort of Neural Network is best suited to predicting a future purchase?
I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4.
Now I'm trying to do a NN that can make a prediction. ...
- 210
3
votes
1
answer
612
views
How to train a LSTM model with multi dimensional data
I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model.
My data is 184 XY coodinates encoded as a numpy ...
- 131
3
votes
0
answers
232
views
What can be considered a deep recurrent neural network?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM.
I have DQN ...
- 61
3
votes
2
answers
80
views
Problem extracting features from convolutional layer where the dimensions are big for feature maps
I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the ...
- 131
3
votes
0
answers
38
views
Does it make sense to add word embeddings as additional features for LSTM model?
I have an LSTM model. This model takes as input tokens. Those tokens represent XML markups extracted from some XML files. My model is working fine. However, I want to optimize it by adding word ...
- 169
2
votes
2
answers
115
views
LSTM exploding? - multiple parallel time series with multiple variables
I have the following situation:
Stock
Time_Stamps
Feature_1
Feature_2
Feature_n
Price
Stock_1
2019
0.5
1.0
1.0
100
Stock_1
2020
0.7
1.3
0.9
90
Stock_2
2019
0.3
0.9
1.1
110
Stock_2
2020
0.2
0.8
1....
- 21
2
votes
0
answers
152
views
Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?
Background
I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the ...
- 71
2
votes
0
answers
96
views
Machine learning with raw data alone / or raw data with its statistics
My question is very general and it does not originate from a specific problem. Let's assume that, through experience, we have learned that some statistical property of a set of data is important in ...
- 61
2
votes
0
answers
44
views
If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?
I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells.
Should I train two models separately or should I just increase ...
- 181
2
votes
0
answers
53
views
Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?
When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN.
Lets say instead of using dense ...
- 2,145
2
votes
0
answers
63
views
How to understand the matrices used in the Attention layer?
Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
- 181
2
votes
0
answers
26
views
Visualisation for Features to Predict Timeseries Data
I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
2
votes
0
answers
32
views
How to feed key-value features (aggregated data) to LSTM?
I have the following time-series aggregated input for an LSTM-based model:
...
- 121
2
votes
2
answers
215
views
How do LSTM and GRU avoid to overcome the vanishing gradient problem?
I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
- 1,513
2
votes
0
answers
177
views
How to implement a LSTM for multilabel classification problem?
I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800.
However, I am not sure of how to classify a certain situation when each input ...
- 31
2
votes
0
answers
174
views
Can the cross-entropy loss be used for a NLP task with LSTM?
I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
- 43
2
votes
0
answers
138
views
How do I make my LSTM model more sensitive to changes in the sequence?
I have a many to one LSTM model for multiclass classification. For reference, this is the architecture of the model
...
- 21
2
votes
0
answers
28
views
Did people analyze dynamics of very simple LSTMs?
I wonder if researchers tried to understand how LSTMs work by analyzing the dynamics of simple LSTM (e.g. with 2 units)? For example, how the hidden state evolves depending on the properties of weight ...
- 21
2
votes
0
answers
30
views
How are batch statistics computed in Recurrent Batch Normalization?
I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, ...
2
votes
1
answer
220
views
How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?
I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
- 181
2
votes
0
answers
128
views
How can I detect fast and slow motion in videos?
I'm trying to detect if a given video shot is fast or slow motion. Basically, I need to calculate a "video motion" score in a given video sequence, meaning how fast or slow motion the video is. For ...
- 983
2
votes
0
answers
167
views
Spike detection in time series using Artificial Neural Networks
I'm quite new in ANNs. I intend to use ANNs for predicting spike points in time series right before they happen. I've already used LSTM for another scenario, and I know that they can be used in ...
- 121
2
votes
0
answers
320
views
Why experience reply memory in DQN instead of a RNN memory?
I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
- 1,016
2
votes
0
answers
57
views
RNN: Different test results on balanced and unbalanced data
I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each ...
- 68
2
votes
0
answers
51
views
Multi-field text input for LSTM
I'm using LSTM to categorize medium-sized pieces of text. Each item to be categorized has several free-form text fields, in addition to several categorical fields. What is the best approach to using ...
- 71
2
votes
0
answers
48
views
How does a neural network output text box location data?
I'm interested in creating a convolutional neural network or LSTM to locate text in an image. I don't want to OCR the text yet, just find the text regions. Yes, I know Tesseract and other systems can ...
2
votes
1
answer
136
views
Price difference predictions curve almost vanished
With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
- 173
2
votes
0
answers
109
views
Combine two embeddding inputs to increase more performance in LSTM model
The situation I encountered here is that I have two inputs(for instance, image embedding, etc.) into the first lstm of a series of lstms to predict the next word to generate sentence(from the second ...
- 21
2
votes
0
answers
80
views
seq2seq vector to letters model
I'm looking to build a sequence-to-sequence model that takes in a 2048-long vector of 1s and 0s as my input and translating it to my known output of (a variable length) 1-20 long characters (ex. ...
1
vote
0
answers
25
views
The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination
I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model:
This model first archived about 0.03 accuracy score and gradually ...
- 11
1
vote
1
answer
39
views
Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models
I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help ...
1
vote
1
answer
54
views
Recognize patterns within random sequences
I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
- 111
1
vote
1
answer
29
views
Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?
I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
- 111
1
vote
0
answers
14
views
SCINet: how does interactive learning work?
i'm having some trouble understanding how does the basic building block of a SCINet works. In the paper the author describes the SCI-block with the following figure:
In which $\phi$, $\theta$, $\eta$ ...
1
vote
0
answers
12
views
Where exactly is permutation happening in equation 5 of the paper "Learning with Sets in Multiple Instance Regression Applied to Remote Sensing"?
I am reading the article Learning with Sets in Multiple Instance Regression Applied to Remote Sensing about creating an embedding which is order-invariant to inputs ($m_{l}$). They referred to order-...
- 23
1
vote
0
answers
34
views
The results are not correct when predicting the future for a very long period of time with LSTM
I am currently using LSTM to try to predict future data in AirPassengers.csv.
This is current code op my Colab (sorry for the comments are Japanese)
https://colab.research.google.com/drive/...
- 13
1
vote
0
answers
93
views
LSTM Recursive Forecast
I am confused about the way the LSTM networks work when forecasting with a horizon that is not finite, but I'm rather searching for a prediction in whatever time in future. In physical terms, I would ...
1
vote
0
answers
12
views
LSTM Forecast Evolution
I have a confusion about the way the LSTM networks work when forecasting with an horizon that is not finite but I'm rather searching for a prediction in whatever time in future. In physical terms I ...
- 11