Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views
1
vote
1answer
12 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
0
votes
0answers
8 views

Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
2
votes
1answer
78 views

Why does the error of my LSTM not decrease after 10 epochs?

Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution. ...
0
votes
1answer
14 views

Text classification of non-equal length texts, should I pad left or right?

Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length. For example, spam filtering on blog article: ...
0
votes
0answers
20 views

Time series forecast for everyday for till a distant future

I have time series data for every single day from last 5 years with seasonal variation and a general increase in trend. This is what my data looks like: And I am trying to predict for every single ...
1
vote
1answer
24 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
0
votes
0answers
23 views

Why doesn't my double deep Q network trained with the same training set give consistent performance?

I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM. My input data is a series of time series data and ...
0
votes
1answer
124 views

How to process data in a data stream for a LSTM

How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes? Training phase: I have trained a LSTM to predict a class out of a ...
0
votes
1answer
20 views

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data. I ...
3
votes
1answer
67 views

What sort of Neural Network is best suited to predicting a future purchase?

I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4. Now I'm trying to do a NN that can make a prediction. ...
1
vote
1answer
42 views

How do GPUs faciliate the training of a Deep Learning Architecture?

I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models. To my understanding, GPUs help in performing independent tasks simultaneously to ...
2
votes
1answer
48 views

What is the difference between LSTM and fully connected LSTM?

I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as FC-LSTM may be seen as a multivariate ...
0
votes
0answers
8 views

Separated LSTMs or a global one for cluster of related features

I have an $n$-dimensional time-series to apply LSTM to, $n$ is the number of features for each time point. These features can be clustered according to their concept, for example $n_1, ..., n_4$ are ...
4
votes
2answers
85 views

Why are GRU and LSTM better than standard RNNs?

It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.
0
votes
0answers
16 views

How to train an Encoder-Decoder LSTM for sequence to sequence prediction?

I have a dataset where for each country there is a name (string) and a multivariate time series (all integers). I am trying to use an Encoder-Decoder LSTM to forecast the next time steps in the time ...
1
vote
0answers
29 views

Do you have to add a dense layer onto the final layer of an LSTM?

If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
0
votes
0answers
27 views

Why in RL function approximators with recurrent structures can learn planning?

In the paper An Investigation of Model-Free Planning the authors use ConvLSTM to learn a planning function. In particular, for each input x_t at time-step ...
0
votes
0answers
12 views
2
votes
1answer
22 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
0
votes
0answers
33 views

How can I combine word2vec with tf-idf to have concatenated features?

I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using x = (state, action)...
1
vote
0answers
22 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
0
votes
0answers
11 views

LSTM for imbalanced panel data

The available tutorials are most focused on time series prediction. I am wondering how shall we prepare the input data when it is an imbalanced data? Here is how data looks like. ...
2
votes
0answers
23 views

Did people analyze dynamics of very simple LSTMs?

I wonder if researchers tried to understand how LSTMs work by analyzing the dynamics of simple LSTM (e.g. with 2 units)? For example how the hidden state evolves depending on the properties of weight ...
1
vote
1answer
341 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
2
votes
0answers
12 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
3
votes
0answers
32 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
0
votes
0answers
11 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
2
votes
1answer
32 views

What's the difference between LSTM and GRU?

The main difference between these two structures lies in the number of gates and their specific roles. The role of the Update gate in the GRU is very similar to the Input and Forget gates in the LSTM. ...
1
vote
0answers
18 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
2
votes
0answers
14 views
5
votes
1answer
697 views

What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, ...
0
votes
0answers
19 views

Which NLP model to use to handle long context?

I'm trying to process product data for an e-commerce platform. The goal is to understand products' size. Just to show you some examples on how messy product dimension description is: ...
1
vote
1answer
43 views
1
vote
1answer
105 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
8
votes
3answers
2k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
0
votes
0answers
31 views

Best language model to do dimension description cleansing/normalization?

I'm working on a web scraper to gather product data. product dimensions are very important to me, but they come in different formats: 32w x 45h x 23d width: 32 inch height: 45 inch depth: 12 inch .....
0
votes
0answers
13 views

Keras word ordering task

I'm trying to solve the word ordering task: given a syntactically unordered sentence, recover the right order of the words. The adopted approach is to transform each sentence in a dependency tree and ...
2
votes
0answers
25 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
2
votes
1answer
24 views

How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
1
vote
1answer
50 views

Is there an LSTM-based unsupervised learning algorithm to label a dataset of curves?

I have a big amount of light curves (image below). I am trying to label the points as signal or background (the signal appears usually periodically, several times, for a given light curve). More ...
2
votes
1answer
50 views

How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
3
votes
0answers
147 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-...
3
votes
0answers
26 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
1
vote
1answer
83 views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
2
votes
0answers
58 views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
2
votes
0answers
61 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
1
vote
1answer
17 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
1
vote
0answers
11 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
2
votes
1answer
93 views

Do I need LSTM units everywhere in the network?

I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and ...