Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
2
votes
0answers
27 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
2
votes
1answer
22 views

Does this diagram represent several LSTMs, or one through several timesteps?

I'm trying to read this paper describing Google's LSTM architecture for machine translation. It features this diagram on page 4: I'm interested in the encoder block, on the left. Apparently, the pink ...
0
votes
0answers
21 views

How does Google's 2016 GNMT architecture work?

I'm trying to read this paper describing Google's LSTM architecture for machine translation from 2016. However, I'm getting stuck as certain things are described too vaguely for me. This is a picture ...
1
vote
0answers
24 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
0
votes
0answers
17 views

How to afine the extremity values in regression prediction with Keras?

I made a stack of bidirectional LSTM layers following by Dense layers (with swish activation functions) in order to predict a continuous value between 0 and 2. I compiled the model with ...
3
votes
1answer
50 views

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information? What ...
1
vote
1answer
14 views

How do LSTMs work if the following two matrices are not able to be multiplied?

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance: The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$ The input data at timestep t ...
1
vote
0answers
42 views

What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?

I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...
0
votes
0answers
23 views

In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?

There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs. Both of these sources mention the reason LSTMs are ...
2
votes
1answer
44 views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
0
votes
0answers
18 views

Stateful many-to-many RNN generating artefacts at regular intervals

I am training a stateful LSTM network, with a time series consisting of about 500000 data points spread over 5 years. This time series is split up to batches of 100 timesteps, and fed into the network....
1
vote
0answers
19 views

Duplicating calculations in CNN-LSTM architecture

I want to use frames from video game and analyze them using CNN and LSTM. But when I have the model defined like that ...
0
votes
1answer
22 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
0
votes
0answers
9 views

What is the difference between TimeDistributed CNN+RNN and passing sequence of features to RNN

I found Keras API that allows to add additional time dimension. ...
1
vote
2answers
44 views

Extract features with CNN and pass as sequence to RNN

I read an article about captioning videos https://blog.coast.ai/five-video-classification-methods-implemented-in-keras-and-tensorflow-99cad29cc0b5 and I want to use solution number 4 (extract features ...
0
votes
0answers
13 views

Is there a model suitable to predict one correct value based on a 2D input series?

I am using an encoder-decoder architecture, with 2 layers each in the encoder and decoder and 128 neurons in each hidden layer. The inputs are in a 2D form: one column has the days and the other ...
0
votes
1answer
42 views

What type of model should I fit to increase accuracy?

Currently, I'm working on 6-axis IMU(Inertial Measurment Unit) dataset. This dataset contain 6 axis IMU data of 7 different drivers. The Imu sensor attached on vehicle. The drivers drives same path. ...
0
votes
1answer
72 views

Feeding YOLOv4 image data into LSTM layer?

How would one extract the feature vector from a given input image using YOLOv4 and pass that data into an LSTM to generate captions for the image? I am trying to make an image captioning software in ...
0
votes
0answers
19 views

Avoiding Overfitting with a large LSTM net on a small amount of data

1. Context I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. There'...
0
votes
0answers
12 views

Python code in LSTM to look at selective history

My dataset has 3 columns - a,b,c. Using b (and its history), I wish to predict c. Using list function and converting to array, I can tell python to look at last 20 b's for any b, as input to predict c....
4
votes
1answer
166 views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
1
vote
0answers
21 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
0
votes
0answers
20 views

Descent Training episodes for LSTM + TD3

I am building an AI with TD3 and lstm in both actor and critic. By LSTM size is 5,5 with 3 layers and hidden layers with 400 and 300 neurons respectively. I have states dimension of 5 with each value ...
0
votes
0answers
20 views

TD3+lstm predicting the same output for varying states

I have a model with TD3 + lstm in both actor and critic. I am trying to make it learn to predict some specific actions based on the environment conditions. However i see that the AI predicts very ...
0
votes
0answers
29 views

Generating artificial data by means of LSTM

I got two classes namely positive and negative with 1500 samples on each a total of 3k. A sample sequence is like: ...
0
votes
1answer
46 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
0answers
33 views

Network design to learn multiple sequences of multiple categories

For learning a single sequence, LSTM only should suffice. However, my situation is different here. I have a list of sequences to learn: The sale volumes of 12 months, these are the sequences And ...
0
votes
0answers
26 views

How to feed the LSTM with different length for the latest time step?

I am having a training data set for a time-series dataset like below: ...
0
votes
1answer
42 views

Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
0
votes
1answer
28 views

Text classification of non-equal length texts, should I pad left or right?

Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length. For example, spam filtering on blog article: ...
0
votes
0answers
30 views

Time series forecast for everyday for till a distant future

I have time series data for every single day from last 5 years with seasonal variation and a general increase in trend. This is what my data looks like: And I am trying to predict for every single ...
0
votes
0answers
34 views

Why doesn't my double deep Q network trained with the same training set give consistent performance?

I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM. My input data is a series of time series data and ...
0
votes
1answer
107 views

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data. I ...
2
votes
1answer
64 views

How do GPUs faciliate the training of a Deep Learning Architecture?

I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models. To my understanding, GPUs help in performing independent tasks simultaneously to ...
0
votes
0answers
17 views

Separated LSTMs or a global one for cluster of related features

I have an $n$-dimensional time-series to apply LSTM to, $n$ is the number of features for each time point. These features can be clustered according to their concept, for example $n_1, ..., n_4$ are ...
2
votes
1answer
114 views

What is the difference between LSTM and fully connected LSTM?

I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as FC-LSTM may be seen as a multivariate ...
0
votes
0answers
30 views

How to train an Encoder-Decoder LSTM for sequence to sequence prediction?

I have a dataset where for each country there is a name (string) and a multivariate time series (all integers). I am trying to use an Encoder-Decoder LSTM to forecast the next time steps in the time ...
1
vote
0answers
73 views

Do you have to add a dense layer onto the final layer of an LSTM?

If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
0
votes
0answers
35 views

Why in RL function approximators with recurrent structures can learn planning?

In the paper An Investigation of Model-Free Planning the authors use ConvLSTM to learn a planning function. In particular, for each input x_t at time-step ...
1
vote
0answers
37 views

LSTM - MAPE Loss Function gives Better Results when Data is De-Scaled before Loss Calculation

I am building an LSTM for predicting a price chart. MAPE resulted in the best loss function compared to ...
2
votes
1answer
41 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
2
votes
0answers
40 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
0
votes
0answers
34 views

LSTM for imbalanced panel data

The available tutorials are most focused on time series prediction. I am wondering how shall we prepare the input data when it is an imbalanced data? Here is how data looks like. ...
1
vote
1answer
25 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
1
vote
1answer
42 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
2
votes
0answers
20 views

Visualisation for Features to Predict Timeseries Data

I have a course assignment to use an LSTM to predict the movement directions of stock prices. One of the things I am asked to do is provide a visualization to compare the predictive powers of a set of ...
3
votes
0answers
44 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
1
vote
0answers
22 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
28 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...

1
2 3 4 5