Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
2
votes
0answers
42 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
0
votes
0answers
45 views

LSTM for imbalanced panel data

The available tutorials are most focused on time series prediction. I am wondering how shall we prepare the input data when it is an imbalanced data? Here is how data looks like. ...
1
vote
1answer
27 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
1
vote
1answer
44 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
2
votes
0answers
21 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
3
votes
0answers
44 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
1
vote
0answers
22 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
28 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
2
votes
1answer
79 views

What's the difference between LSTM and GRU?

The main difference between these two structures lies in the number of gates and their specific roles. The role of the Update gate in the GRU is very similar to the Input and Forget gates in the LSTM. ...
2
votes
0answers
27 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
1
vote
0answers
30 views

Which NLP model to use to handle long context?

I'm trying to process product data for an e-commerce platform. The goal is to understand products' size. Just to show you some examples on how messy product dimension description is: ...
0
votes
0answers
37 views

Best language model to do dimension description cleansing/normalization?

I'm working on a web scraper to gather product data. product dimensions are very important to me, but they come in different formats: 32w x 45h x 23d width: 32 inch height: 45 inch depth: 12 inch .....
0
votes
0answers
20 views

Keras word ordering task

I'm trying to solve the word ordering task: given a syntactically unordered sentence, recover the right order of the words. The adopted approach is to transform each sentence in a dependency tree and ...
4
votes
0answers
64 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
3
votes
0answers
34 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
1
vote
1answer
709 views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
20
votes
3answers
11k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
2
votes
0answers
30 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
1
vote
1answer
26 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
2
votes
0answers
71 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
2
votes
1answer
193 views

Is this Keras LSTM model underfitting?

I think this model is underfitting. Is this correct? ...
1
vote
0answers
74 views

Do RNN solves the need for LSTM and/or multiple states in Deep Q-Learning?

Introduction I am trying to setup a Deep Q-Learning agent. I have looked that the papers Playing Atari with Deep Reinforcement Learning as well as Deep Recurrent Q-Learning for Partially Observable ...
1
vote
0answers
25 views

length independent sequence classification methods

I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good ...
1
vote
0answers
35 views

Building a template based NLG system to generate a report from data

I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several ...
3
votes
0answers
29 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
3
votes
1answer
135 views

What are pros and cons of Bi-LSTM as compared to LSTM?

What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
3
votes
0answers
36 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
2
votes
3answers
149 views

Using sigmoid in LSTM network for multi-step forecasting

I'm trying to develop a multistep forecasting model using LSTM Network. The model takes three times steps as input and predicting two time_steps. both input and output columns are normalised using ...
2
votes
1answer
61 views

How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
1
vote
0answers
42 views

Model for supervised sequence classification task

The Problem I am currently working on a sequence classification problem I try to solve with machine learning. The target variable is the current state of a system. This target variable is following a ...
2
votes
0answers
39 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
1
vote
1answer
57 views

Are sentences from the same document independent and identically distributed?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1, ...
1
vote
0answers
33 views

How should I go about selecting an optimal num_units within a LSTM cell for different sequence sizes

I am currently working on a stock market prediction model which incorporates sentiments along with historical price for next day price prediction. I wanted to test different window / sequence size e....
0
votes
0answers
61 views

My CTC loss model's loss stagnates and then outputs only blank characters

I am trying to implement BaiDu's DeepSpeech1 in keras using CTC loss, my code is below: ...
1
vote
0answers
29 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
1
vote
0answers
21 views

Recurrent neural Network for survival analyses: Dealing with forecast data as feature which can exceed the number of days untill a event occurs

I am building a Recurrent Neural network (LSTM) for predicting the number of days until a Pollen season starts (when the cumulative of the year exceeds X). One of the features I am including in my ...
1
vote
0answers
29 views

Feed data into Keras LSTM layer [closed]

I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it. I've ...
1
vote
1answer
2k views

What’s the difference between LSTM and RNN?

What's the difference between LSTM and RNN? I know that RNN is a network layer used in neural networks, but what exactly is an LSTM? Is it also a network layer with the same characteristics?
1
vote
0answers
31 views

Can tensorflow debugger debug a trained keras model during prediction?

I have a trained LSTM model created with keras. Is it possible to use tensorboard debugger or tensorflow debugger to debug the model during prediction runtime? Meaning that it steps through the model ...
1
vote
0answers
21 views

When stacking LSTM's, should the hidden units increase?

I'm using Weights and Biases to do some hyperparameter sweeping for a supervised sequence-to-sequence problem I'm working on. One thing I noticed is that the sweeps with a gradually increasing number ...
1
vote
0answers
45 views

Simple sequential model with LSTM which doesn't converge

I'm actually trying to create a sequential neural network in order to translate a "human" sentence in a "machine" sentence understandable by an algorithm. Like It didn't work, I've try to create a NN ...
1
vote
1answer
47 views

time-series prediction : loss going down, then stagnates with very high variance

I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in [0,13]. I have noticed that one-hot encoding it and ...
1
vote
0answers
55 views

Are there any tools I can use to debug a Keras LSTM model during prediction?

I want to be able to debug my Keras LSTM model. For example, I want to be able to check the values of the input/output gates, cell states and hidden states at every time-step during prediction. Are ...
0
votes
0answers
15 views

What will be the sequence of steps in a human activity recognition model using LSTM?

In the context of these steps detection, tracking, action classification and activity recognition. Which step will be first and further sequence?
0
votes
0answers
24 views

RNN/LSTM with a large amounts of data

I have sequence data that's quite large - 4x65k per sample. I'm interested in doing classification problems. The number of classes is moderate - ~27 or so What is the suggestion for dealing with this ...
2
votes
0answers
43 views

How do I make my LSTM model more sensitive to changes in the sequence?

I have a many to one LSTM model for multiclass classification. For reference, this is the architecture of the model ...
3
votes
1answer
66 views

LSTM model on different time scales

I am a newbie to machine learning. I have an LSTM model that predicts the next output n+1 time 1, params 1, output 1 time 2, params 2, output 2 time 3, params 3, output 3 . . time n, params n, , ...
0
votes
0answers
28 views

Which hidden state should I use for a trajectory when incorporating LSTM into RL?

I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited ...
0
votes
1answer
58 views

LSTM implementation in KERAS [closed]

I would like to build an LSTM to predict the correct words order given a sentence. My dataset is composed of sentences, where each sentence has a variable number of words (each word is embedded). The ...
4
votes
1answer
225 views

Is the LSTM component a neuron or a layer?

Given the standard illustrative feed-forward neural net model, with the dots as neurons and the lines as neuron-to-neuron connection, what part is the (unfold) LSTM cell (see picture)? Is it a neuron (...