Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
0
votes
1answer
140 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
0answers
34 views

Network design to learn multiple sequences of multiple categories

For learning a single sequence, LSTM only should suffice. However, my situation is different here. I have a list of sequences to learn: The sale volumes of 12 months, these are the sequences And ...
0
votes
1answer
185 views

Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
0
votes
1answer
42 views

Text classification of non-equal length texts, should I pad left or right?

Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length. For example, spam filtering on blog article: ...
1
vote
1answer
273 views

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data. I ...
2
votes
1answer
69 views

How do GPUs faciliate the training of a Deep Learning Architecture?

I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models. To my understanding, GPUs help in performing independent tasks simultaneously to ...
2
votes
1answer
672 views

What is the difference between LSTM and fully connected LSTM?

I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as FC-LSTM may be seen as a multivariate ...
1
vote
0answers
304 views

Do you have to add a dense layer onto the final layer of an LSTM?

If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
1
vote
0answers
81 views

LSTM - MAPE Loss Function gives Better Results when Data is De-Scaled before Loss Calculation

I am building an LSTM for predicting a price chart. MAPE resulted in the best loss function compared to ...
2
votes
1answer
48 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
2
votes
0answers
46 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
1
vote
1answer
37 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
1
vote
1answer
53 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
2
votes
0answers
22 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
3
votes
0answers
46 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
1
vote
0answers
25 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
32 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
4
votes
1answer
336 views

What's the difference between LSTM and GRU?

I have been reading about LSTMs and GRUs, which are recurrent neural networks (RNNs). The difference between the two is the number and specific type of gates that they have. The GRU has an update gate,...
2
votes
0answers
28 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
1
vote
0answers
34 views

Which NLP model to use to handle long context?

I'm trying to process product data for an e-commerce platform. The goal is to understand products' size. Just to show you some examples on how messy product dimension description is: ...
4
votes
0answers
85 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
3
votes
0answers
40 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
1
vote
1answer
3k views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
35
votes
3answers
27k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
2
votes
1answer
71 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
1
vote
1answer
28 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
2
votes
0answers
89 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
1
vote
1answer
398 views

Is this LSTM model underfitting?

I think this model is underfitting. Is this correct? ...
1
vote
0answers
104 views

Do RNN solves the need for LSTM and/or multiple states in Deep Q-Learning?

Introduction I am trying to setup a Deep Q-Learning agent. I have looked that the papers Playing Atari with Deep Reinforcement Learning as well as Deep Recurrent Q-Learning for Partially Observable ...
1
vote
0answers
27 views

length independent sequence classification methods

I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good ...
1
vote
0answers
128 views

Building a template based NLG system to generate a report from data

I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several ...
3
votes
0answers
31 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
4
votes
1answer
260 views

What are pros and cons of Bi-LSTM as compared to LSTM?

What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
3
votes
0answers
37 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
2
votes
3answers
176 views

Using sigmoid in LSTM network for multi-step forecasting

I'm trying to develop a multistep forecasting model using LSTM Network. The model takes three times steps as input and predicting two time_steps. both input and output columns are normalised using ...
2
votes
1answer
150 views

How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
1
vote
0answers
47 views

Model for supervised sequence classification task

The Problem I am currently working on a sequence classification problem I try to solve with machine learning. The target variable is the current state of a system. This target variable is following a ...
2
votes
0answers
60 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
1
vote
1answer
58 views

Are sentences from the same document independent and identically distributed?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1, ...
1
vote
0answers
66 views

How should I go about selecting an optimal num_units within a LSTM cell for different sequence sizes

I am currently working on a stock market prediction model which incorporates sentiments along with historical price for next day price prediction. I wanted to test different window / sequence size e....
1
vote
0answers
37 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
1
vote
0answers
23 views

Recurrent neural Network for survival analyses: Dealing with forecast data as feature which can exceed the number of days untill a event occurs

I am building a Recurrent Neural network (LSTM) for predicting the number of days until a Pollen season starts (when the cumulative of the year exceeds X). One of the features I am including in my ...
1
vote
0answers
31 views

Feed data into Keras LSTM layer [closed]

I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it. I've ...
1
vote
1answer
2k views

What’s the difference between LSTM and RNN?

What's the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
1
vote
0answers
31 views

Can tensorflow debugger debug a trained keras model during prediction?

I have a trained LSTM model created with keras. Is it possible to use tensorboard debugger or tensorflow debugger to debug the model during prediction runtime? Meaning that it steps through the model ...
1
vote
0answers
25 views

When stacking LSTM's, should the hidden units increase?

I'm using Weights and Biases to do some hyperparameter sweeping for a supervised sequence-to-sequence problem I'm working on. One thing I noticed is that the sweeps with a gradually increasing number ...
1
vote
0answers
60 views

Simple sequential model with LSTM which doesn't converge

I'm actually trying to create a sequential neural network in order to translate a "human" sentence in a "machine" sentence understandable by an algorithm. Like It didn't work, I've try to create a NN ...
1
vote
1answer
66 views

time-series prediction : loss going down, then stagnates with very high variance

I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in [0,13]. I have noticed that one-hot encoding it and ...
1
vote
0answers
87 views

Are there any tools I can use to debug a Keras LSTM model during prediction?

I want to be able to debug my Keras LSTM model. For example, I want to be able to check the values of the input/output gates, cell states and hidden states at every time-step during prediction. Are ...
2
votes
0answers
73 views

How do I make my LSTM model more sensitive to changes in the sequence?

I have a many to one LSTM model for multiclass classification. For reference, this is the architecture of the model ...