Questions tagged [long-short-term-memory]
For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".
298
questions
1
vote
1
answer
94
views
Recommended Time serie forecasting model for Fibonacci levels classification
I have a set of time series data which gives me fibonacci levels and the duration at which the value is at this level.
Data structure to look like:
...
- 13
0
votes
1
answer
1k
views
Seq2Seq model produces repeating words
My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
0
votes
1
answer
171
views
Advantages of CNN vs. LSTM for sequence data like text or log-files
When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what ...
- 1
1
vote
1
answer
105
views
How to train an LSTM to classify based on rare historic event?
I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far.
My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle ...
- 759
0
votes
1
answer
83
views
Must all CNNs and RNNs not have a fully connected layer in order to be considered as such?
In the paper Wrist-worn blood pressure tracking in healthy free-living individuals using neural networks, the authors talk about a combination of feed-forward and recurrent layers, as if FC layers ...
- 73
1
vote
1
answer
61
views
Do RNNs/LSTMs really need to be sequential?
There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly ...
- 13
1
vote
1
answer
84
views
Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?
RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
- 45
0
votes
1
answer
87
views
How can Image Caption work?
I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the ...
- 111
0
votes
1
answer
159
views
Convert LSTM univariate Autoencoder to multivariate Autoencoder
I have the following code snippet which takes in a single column of value i.e. 1 feature. How do I modify the LSTM model such that it accepts 3 features?
...
- 21
1
vote
2
answers
346
views
What is the input to the left most LSTM cell c(t-1) and h(t-1)?
Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?
- 21
2
votes
1
answer
137
views
Is this LSTM layer learning anything?
I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:
I don't understand. Is this layer learning anything? Or no?
...
1
vote
1
answer
166
views
How to define a "don't care" class in time series classification in Pytorch?
This is a theoretical question.
Setup
I have a time series classification task in which I should output a classification of 3 classes for every time stamp t.
All ...
- 759
1
vote
0
answers
46
views
Given the word embeddings, how do I create the sentence composed of the corresponding words?
I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it).
In any given ...
- 11
2
votes
0
answers
50
views
If I want to predict two unrelated values given the same sequence of data points, should I have a model with two outputs or two models?
I want to predict two separate y-values (not really logically connected) based on an input sequence of data (values x). Using LSTM cells.
Should I train two models separately or should I just increase ...
- 181
1
vote
0
answers
47
views
How to improve prediction performance of periodic data?
I have a 1 column dataset of $50 000$ points where 95% of the values equal $-50$. The data looks like the following: $$\begin{matrix}
\text{time} & \text{value}\\
1&-50 \\
2&-50 \\
3&-...
- 11
0
votes
1
answer
85
views
Is it possible to predict $x^2$, $\log(x)$, or variable function of $x$ using RNN?
There were some posts that using RNN can predict the next point of the sine wave function with data history.
However, I wondered if it also works on all the functions of $x$, such as $x^2$, $x^3$, $\...
0
votes
0
answers
586
views
How to make an ensemble model of two LSTM models with different window sizes i.e. different data shapes
Below is the Python code for making an ensemble model. All the inputs are the same for all three models. But what if the models have different input shapes due to different window size, such as LSTM ...
- 1
1
vote
0
answers
48
views
Using LSTM model to train spatial inputs
I have an $x$-$y$ plane, inside that plane I have 9 paths $(p_1, p_2, \dots, p_3)$. Each path is classified into one of the three classes $(c_1, c_2, c_3)$. Each path has 100 coordinates points i.e $((...
- 111
8
votes
3
answers
10k
views
How should we regularize an LSTM model?
There are five parameters from an LSTM layer for regularization if I am correct.
To deal with overfitting, I would start with
reducing the layers
reducing the hidden units
Applying dropout or ...
- 133
0
votes
1
answer
2k
views
Is the working of RNNs, LSTM and GRU sequential or parallel?
You take any blog or any example and all they tell you about is the given picture below.
It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in ...
- 263
0
votes
0
answers
21
views
Is it possible and if so does it make sense to have dense layers in between LSTM layers?
I am new to LSTMs and I was wondering if it is possible to have LSTM layer then dense then LSTM again and does it make sense?
- 1
0
votes
1
answer
748
views
Why won't my model train with CTC loss?
I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy ...
- 101
0
votes
0
answers
302
views
Using an LSTM for model-based RL in a POMDP
I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
- 1
2
votes
1
answer
76
views
Understanding LSTM through example
I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states.
Lets assume that ...
- 121
0
votes
3
answers
838
views
Time series prediction using LSTM and CNN-LSTM: which is better?
I am working on LSTM and CNN to solve the time series prediction problem.
I have seen some tutorial examples of time series prediction using CNN-LSTM. But I don't know if it is better than what I ...
- 17
2
votes
0
answers
104
views
Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?
When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN.
Lets say instead of using dense ...
- 2,890
2
votes
1
answer
71
views
Does this diagram represent several LSTMs, or one through several timesteps?
I'm trying to read this paper describing Google's LSTM architecture for machine translation. It features this diagram on page 4:
I'm interested in the encoder block, on the left. Apparently, the pink ...
- 262
1
vote
0
answers
173
views
How is input defined for a biaxial lstm network for generating music?
I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
- 111
3
votes
1
answer
81
views
Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?
Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information?
What ...
- 165
1
vote
1
answer
83
views
How do LSTMs work if the following two matrices are not able to be multiplied?
In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:
The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$
The input data at timestep t ...
- 177
1
vote
0
answers
2k
views
What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?
I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...
- 51
0
votes
0
answers
68
views
In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?
There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs.
Both of these sources mention the reason LSTMs are ...
- 177
3
votes
1
answer
2k
views
Any comparison between transformer and RNN+Attention on the same dataset?
I am wondering what is believed to be the reason for superiority of transformer?
I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
1
vote
1
answer
58
views
Appropriate metric and approach for natural language generation for small sentences
I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
- 326
2
votes
2
answers
734
views
Extract features with CNN and pass as sequence to RNN
I read an article about captioning videos and I want to use solution number 4 (extract features with a CNN, pass the sequence to a separate RNN) in my own project.
But for me, it seems really strange ...
user40943
0
votes
1
answer
56
views
What type of model should I fit to increase accuracy?
Currently, I'm working on 6-axis IMU(Inertial Measurment Unit) dataset. This dataset contain 6 axis IMU data of 7 different drivers. The Imu sensor attached on vehicle. The drivers drives same path. ...
- 257
0
votes
1
answer
623
views
Feeding YOLOv4 image data into LSTM layer?
How would one extract the feature vector from a given input image using YOLOv4 and pass that data into an LSTM to generate captions for the image?
I am trying to make an image captioning software in ...
- 11
1
vote
1
answer
3k
views
36
votes
2
answers
20k
views
How can Transformers handle arbitrary length input?
The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
- 2,890
1
vote
0
answers
77
views
When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?
I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this.
Now, I want to know that for a problem/game like Breakout, where ...
- 121
0
votes
1
answer
295
views
What is the big fuzz about SHA-RNN versus Transformers?
In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
- 193
0
votes
0
answers
37
views
Network design to learn multiple sequences of multiple categories
For learning a single sequence, LSTM only should suffice.
However, my situation is different here. I have a list of sequences to learn:
The sale volumes of 12 months, these are the sequences
And ...
- 1,283
0
votes
1
answer
1k
views
Number of LSTM layers needed to learn a certain number of sequences
Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.
For example, in Keras:
...
- 1,283
0
votes
1
answer
91
views
Text classification of non-equal length texts, should I pad left or right?
Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length.
For example, spam filtering on blog article:
...
- 1,283
1
vote
1
answer
596
views
How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.
I ...
- 31
2
votes
1
answer
148
views
How do GPUs faciliate the training of a Deep Learning Architecture?
I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.
To my understanding, GPUs help in performing independent tasks simultaneously to ...
3
votes
1
answer
3k
views
What is the difference between LSTM and fully connected LSTM?
I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as
FC-LSTM may be seen as a multivariate ...
- 131
1
vote
0
answers
840
views
Do you have to add a dense layer onto the final layer of an LSTM?
If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
- 4,900
1
vote
0
answers
364
views
LSTM - MAPE Loss Function gives Better Results when Data is De-Scaled before Loss Calculation
I am building an LSTM for predicting a price chart. MAPE resulted in the best loss function compared to ...
2
votes
1
answer
130
views
How do LSTM or GRU gates learn to specialize in their desired tasks?
While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
- 23