Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

125 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
0 answers
178 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
musako's user avatar
  • 181
4 votes
0 answers
127 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
pairon's user avatar
  • 143
4 votes
0 answers
283 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
Kornephoros's user avatar
4 votes
0 answers
115 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
adamconkey's user avatar
4 votes
4 answers
1k views

Use Machine/Deep Learning to Guess a String

I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc). ...
TreHoffman's user avatar
3 votes
0 answers
112 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
Pluviophile's user avatar
  • 1,263
3 votes
0 answers
51 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
ADSBJason's user avatar
3 votes
0 answers
41 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
Erick Medina's user avatar
3 votes
0 answers
105 views

Why don't the neural networks inside LSTM cells contain hidden layers?

I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers. Why don't LSTM cells have more complex neural ...
Snowybluesky's user avatar
3 votes
0 answers
163 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
Yining's user avatar
  • 31
3 votes
0 answers
88 views

Why do regression LSTMs learn high to low inputs significantly better than low to high?

The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361. I have ...
Recessive's user avatar
  • 1,386
3 votes
0 answers
100 views

What sort of Neural Network is best suited to predicting a future purchase?

I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4. Now I'm trying to do a NN that can make a prediction. ...
NeomerArcana's user avatar
3 votes
1 answer
710 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
Dawid's user avatar
  • 131
3 votes
0 answers
253 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
Savco's user avatar
  • 61
3 votes
2 answers
88 views

Problem extracting features from convolutional layer where the dimensions are big for feature maps

I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the ...
I. A's user avatar
  • 131
3 votes
0 answers
39 views

Does it make sense to add word embeddings as additional features for LSTM model?

I have an LSTM model. This model takes as input tokens. Those tokens represent XML markups extracted from some XML files. My model is working fine. However, I want to optimize it by adding word ...
Emna Jaoua's user avatar
2 votes
2 answers
129 views

LSTM exploding? - multiple parallel time series with multiple variables

I have the following situation: Stock Time_Stamps Feature_1 Feature_2 Feature_n Price Stock_1 2019 0.5 1.0 1.0 100 Stock_1 2020 0.7 1.3 0.9 90 Stock_2 2019 0.3 0.9 1.1 110 Stock_2 2020 0.2 0.8 1....
Sphenoidale's user avatar
2 votes
0 answers
173 views

Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?

Background I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the ...
jumelet's user avatar
  • 71
2 votes
0 answers
117 views

Machine learning with raw data alone / or raw data with its statistics

My question is very general and it does not originate from a specific problem. Let's assume that, through experience, we have learned that some statistical property of a set of data is important in ...
user101464's user avatar
2 votes
0 answers
49 views

If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?

I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells. Should I train two models separately or should I just increase ...
Jake B.'s user avatar
  • 181
2 votes
0 answers
98 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
chessprogrammer's user avatar
2 votes
0 answers
86 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
Joe Black's user avatar
  • 181
2 votes
0 answers
30 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
georgi koyrushki's user avatar
2 votes
0 answers
37 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
Maximus's user avatar
  • 121
2 votes
2 answers
337 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
DRV's user avatar
  • 1,673
2 votes
0 answers
233 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
mgb's user avatar
  • 31
2 votes
0 answers
205 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
Leey's user avatar
  • 43
2 votes
0 answers
153 views

How do I make my LSTM model more sensitive to changes in the sequence?

I have a many to one LSTM model for multiclass classification. For reference, this is the architecture of the model ...
PotatoCoding's user avatar
2 votes
0 answers
28 views

Did people analyze dynamics of very simple LSTMs?

I wonder if researchers tried to understand how LSTMs work by analyzing the dynamics of simple LSTM (e.g. with 2 units)? For example, how the hidden state evolves depending on the properties of weight ...
bastak's user avatar
  • 21
2 votes
0 answers
31 views

How are batch statistics computed in Recurrent Batch Normalization?

I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, ...
OverLordGoldDragon's user avatar
2 votes
1 answer
253 views

How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
Asif Khan's user avatar
  • 181
2 votes
0 answers
182 views

How can I detect fast and slow motion in videos?

I'm trying to detect if a given video shot is fast or slow motion. Basically, I need to calculate a "video motion" score in a given video sequence, meaning how fast or slow motion the video is. For ...
Tina J's user avatar
  • 973
2 votes
0 answers
190 views

Spike detection in time series using Artificial Neural Networks

I'm quite new in ANNs. I intend to use ANNs for predicting spike points in time series right before they happen. I've already used LSTM for another scenario, and I know that they can be used in ...
Ehsan Khodarahmi's user avatar
2 votes
0 answers
391 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
Eka's user avatar
  • 1,066
2 votes
0 answers
59 views

RNN: Different test results on balanced and unbalanced data

I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each ...
ketzul's user avatar
  • 68
2 votes
0 answers
52 views

Multi-field text input for LSTM

I'm using LSTM to categorize medium-sized pieces of text. Each item to be categorized has several free-form text fields, in addition to several categorical fields. What is the best approach to using ...
Derek Hans's user avatar
2 votes
0 answers
49 views

How does a neural network output text box location data?

I'm interested in creating a convolutional neural network or LSTM to locate text in an image. I don't want to OCR the text yet, just find the text regions. Yes, I know Tesseract and other systems can ...
Matthew Bishop's user avatar
2 votes
1 answer
147 views

Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
user1050421's user avatar
2 votes
0 answers
111 views

Combine two embeddding inputs to increase more performance in LSTM model

The situation I encountered here is that I have two inputs(for instance, image embedding, etc.) into the first lstm of a series of lstms to predict the next word to generate sentence(from the second ...
user14947's user avatar
2 votes
0 answers
82 views

seq2seq vector to letters model

I'm looking to build a sequence-to-sequence model that takes in a 2048-long vector of 1s and 0s as my input and translating it to my known output of (a variable length) 1-20 long characters (ex. ...
rajkarthikkumar's user avatar
1 vote
0 answers
44 views

The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination

I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model: This model first archived about 0.03 accuracy score and gradually ...
Đạt Trần's user avatar
1 vote
1 answer
66 views

Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models

I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help ...
Guilherme Vieira's user avatar
1 vote
1 answer
82 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
FELIPE_RIBAS's user avatar
1 vote
1 answer
137 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
ad124j2's user avatar
  • 11
1 vote
1 answer
40 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
Krusty the Clown's user avatar
1 vote
0 answers
22 views

SCINet: how does interactive learning work?

i'm having some trouble understanding how does the basic building block of a SCINet works. In the paper the author describes the SCI-block with the following figure: In which $\phi$, $\theta$, $\eta$ ...
Juan Hirschmann's user avatar
1 vote
0 answers
29 views

bad prediction when having noise on the data: LSTM time-series regression

I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
stack offer's user avatar
1 vote
0 answers
14 views

Where exactly is permutation happening in equation 5 of the paper "Learning with Sets in Multiple Instance Regression Applied to Remote Sensing"?

I am reading the article Learning with Sets in Multiple Instance Regression Applied to Remote Sensing about creating an embedding which is order-invariant to inputs ($m_{l}$). They referred to order-...
Oculu 's user avatar
  • 43
1 vote
0 answers
110 views

How is it possible to use batches of data from within the same sequence with an LSTM?

ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless? Using the example of an LSTM, it has a hidden ...
Recessive's user avatar
  • 1,386
1 vote
0 answers
39 views

The results are not correct when predicting the future for a very long period of time with LSTM

I am currently using LSTM to try to predict future data in AirPassengers.csv. This is current code op my Colab (sorry for the comments are Japanese) https://colab.research.google.com/drive/...
k_ele's user avatar
  • 13