Skip to main content

Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
73 votes
4 answers
104k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
DRV's user avatar
  • 1,703
68 votes
4 answers
122k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
Stephen Johnson's user avatar
38 votes
2 answers
22k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
chessprogrammer's user avatar
25 votes
4 answers
9k views

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
shiredude95's user avatar
20 votes
4 answers
30k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
Ahsan Tarique's user avatar
18 votes
1 answer
19k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
Kevin. Fang's user avatar
16 votes
4 answers
30k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user8714896's user avatar
14 votes
1 answer
5k views

Which approaches could I use to create a simple chatbot using a neural network?

I wanted to start experimenting with neural networks, so I decided to make a chatbot (like Cleverbot, which is not that clever anyway) using them. I looked around for some documentation and I found ...
Totem's user avatar
  • 381
14 votes
3 answers
7k views

What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, ...
kuixiong's user avatar
  • 241
11 votes
1 answer
2k views

How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM? So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), ...
user8714896's user avatar
10 votes
3 answers
4k views

Should I choose a model with the smallest loss or highest accuracy?

I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data): Model A: Accuracy: ~91%, Loss: ~0.01 Model B: Accuracy: ~83%, Loss: ~0.003 The ...
malioboro's user avatar
  • 2,819
8 votes
2 answers
21k views

Can LSTM neural networks be sped up by a GPU?

I am training LSTM neural networks with Keras on a small mobile GPU. The speed on the GPU is slower than on the CPU. I found some articles that say that it is hard to train LSTMs (and, in general, ...
Dieshe's user avatar
  • 289
8 votes
2 answers
2k views

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs ...
SuperCodeBrah's user avatar
8 votes
1 answer
29k views

What is the difference between LSTM and RNN?

What is the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
Mao76's user avatar
  • 83
8 votes
3 answers
10k views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
Leo's user avatar
  • 133
7 votes
1 answer
1k views

Is LSTM a subcategory of RNN?

Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different? Literature doesn't seem to be unitary on this. This figure appears to explain the models to be alternatives, but I ...
MrPlanck's user avatar
  • 113
6 votes
1 answer
3k views

What are pros and cons of Bi-LSTM as compared to LSTM?

What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
DRV's user avatar
  • 1,703
6 votes
2 answers
406 views

Why are GRU and LSTM better than standard RNNs?

It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.
Deep Analytics's user avatar
6 votes
2 answers
232 views

How to shorten the development time of a neural network?

I am developing an LSTM for sequence tagging. During the development, I do various changes in the system, for example, add new features, change the number of nodes in the hidden layers, etc. After ...
Erel Segal-Halevi's user avatar
6 votes
1 answer
170 views

How are LSTM's trained for text generation?

I've seen some articles about text generation using LSTMs (or GRUs) for text generation. Basically it seems you train them by folding them out, and putting a letter in each input. But say you trained ...
zooby's user avatar
  • 2,216
6 votes
1 answer
2k views

What's the difference between LSTM and GRU?

I have been reading about LSTMs and GRUs, which are recurrent neural networks (RNNs). The difference between the two is the number and specific type of gates that they have. The GRU has an update gate,...
Pluviophile's user avatar
  • 1,273
6 votes
3 answers
3k views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
Asif Khan's user avatar
  • 181
6 votes
0 answers
184 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
musako's user avatar
  • 181
5 votes
1 answer
1k views

Is the LSTM component a neuron or a layer?

Given the standard illustrative feed-forward neural net model, with the dots as neurons and the lines as neuron-to-neuron connection, what part is the (unfold) LSTM cell (see picture)? Is it a neuron (...
MScott's user avatar
  • 445
5 votes
1 answer
1k views

Over- and underestimations of the lowest and highest values in LSTM network

I'm training an LSTM network with multiple inputs and several LSTM layers in order to set up a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation ...
Kristof's user avatar
  • 61
4 votes
2 answers
245 views

Price Movement Forecasting Issue

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
user1050421's user avatar
4 votes
1 answer
883 views

Why do we need multiple LSTM units in a layer?

What is the point of having multiple LSTM units in a single layer? Surely if we have a single unit it should be able to capture (remember) all the data anyway and using more units in the same layer ...
Dema Ushchapovskyy's user avatar
4 votes
2 answers
591 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-...
chausies's user avatar
  • 150
4 votes
1 answer
67 views

Training an RNN to answer simple quesitons

I would like to train an RNN to follow the sentences: "Would you like some cheese"? with "Yes, I would like some cheese." So whenever the template "Would you like some ____?" appears then RNN ...
zooby's user avatar
  • 2,216
4 votes
2 answers
146 views

Using a neural network to identify a stable region within a set of data?

I am working on a problem in which I am attempting to find a stable region in a spiral galaxy. The PI I'm working with asked me to use machine learning as a tool to solve the problem. I have created ...
Pomegranate Society's user avatar
4 votes
1 answer
329 views

How should I design the LSTM architecture for multivariate time series forecasting problems?

There is plenty of literature describing LSTMs in a lot of detail and how to use them for multi-variate or uni-variate forecasting problems. What I couldn't find though, is any papers or discussions ...
Dema Ushchapovskyy's user avatar
4 votes
1 answer
348 views

Object IN/OUT counting using CNN+RNN

I am building a video analytics program for counting moving things in a video. I am detecting bicycles and nothing else. I run object detection using the SSD mobile-net model in all the frames and ...
55597's user avatar
  • 71
4 votes
1 answer
334 views

What would be the best approach to teach an AI to learn how to "sing" along a beat?

I have heard and read about HyperGAN, LSTM and a few other techniques, but I have a hard time piecing the overall concept together. End Goal Being able to input an instrumental and get an output of ...
vaid's user avatar
  • 141
4 votes
0 answers
131 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
pairon's user avatar
  • 143
4 votes
0 answers
294 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
Kornephoros's user avatar
4 votes
0 answers
116 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
adamconkey's user avatar
4 votes
4 answers
1k views

Use Machine/Deep Learning to Guess a String

I want to be able to input a block of text and then have it guess a string within a predefined range (i.e. a string that starts with three letters and ends with five numbers like "XXX12345", etc). ...
TreHoffman's user avatar
3 votes
2 answers
2k views

Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
BlueSnake's user avatar
3 votes
3 answers
5k views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
MASTER OF CODE's user avatar
3 votes
2 answers
729 views

How to use LSTM to generate a paragraph

A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the ...
Dan D.'s user avatar
  • 1,293
3 votes
1 answer
3k views

What is the difference between ConvLSTM and CNN LSTM?

What will be the difference when used for video classification? Will they yield different results or are they the same fundamentally?
user239457's user avatar
3 votes
2 answers
1k views

Does LSTM provide any unique value or advantages compared to other algorithms, including "vanilla" RNN?

I have heard a lot of hype around LSTM for all kinds of time-series based applications including NLP. Despite this, I haven't seen many (if any) applications of LSTM where LSTM performs uniquely well ...
Vladimir Belik's user avatar
3 votes
1 answer
81 views

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information? What ...
ADA's user avatar
  • 165
3 votes
1 answer
2k views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
milad aghajohari's user avatar
3 votes
1 answer
1k views

What is a state in a recurrent neural network?

I am Reading "Supervised Sequence Labelling with Recurrent Neural Networks" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations. Specifically, what I ...
Stephen Johnson's user avatar
3 votes
1 answer
430 views

Can dropout layers not influence LSTM training?

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows: ...
GKozinski's user avatar
  • 1,270
3 votes
1 answer
779 views

Can non-sequential deep learning models outperform sequential models in time series forecasting?

Can a CNN (or other non-sequential deep learning models) outperform LSTM (or other sequential models) in time series data? I know this question is not very specific, but I experienced this when ...
GiorgosMaragkopoulos's user avatar
3 votes
1 answer
930 views

How to change the backward pass for an LSTM layer that outputs to another LSTM layer?

I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far. On pg. 21 of the pdf (pg. ...
Angle Qian's user avatar
3 votes
2 answers
138 views

Why can't LSTMs keep track of the "important parts" of a sequence?

I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format. For ...
user8714896's user avatar
3 votes
1 answer
73 views

When working with time-series data, is it wrong to use different time-steps for the features and target?

When working with time-series data, is it wrong to use daily prices as features and the price after 3 days as the target? Or should I use the next-day price as a target, and, after training, predict 3 ...
George's user avatar
  • 31

1
2 3 4 5
7