Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
0
votes
0answers
80 views

How to make an ensemble model of two LSTM models with different window sizes i.e. different data shapes

Below is the Python code for making an ensemble model. All the inputs are the same for all three models. But what if the models have different input shapes due to different window size, such as LSTM ...
1
vote
1answer
77 views

How to understand marginal loglikelihood objective function as loss function (explanation of an article)?

I am reading article https://allenai.org/paper-appendix/emnlp2017-wt/ http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf about training neural network and the loss function is mentioned ...
0
votes
0answers
99 views

How are temporal links made between following sequences in RNN?

Say I use an RNN, whatever is the cell's type, to perform time series classification. It can thus be seen as sequence classification. The time series is split into random, equal size, overlapping ...
0
votes
0answers
53 views

Why in RL function approximators with recurrent structures can learn planning?

In the paper An Investigation of Model-Free Planning, the authors use ConvLSTM to learn a planning function. In particular, for each input $x_t$ at time-step $t$, the function approximator is run for ...
12
votes
1answer
5k views

Which approaches could I use to create a simple chatbot using a neural network?

I wanted to start experimenting with neural networks, so I decided to make a chatbot (like Cleverbot, which is not that clever anyway) using them. I looked around for some documentation and I found ...
8
votes
3answers
2k views

Should I choose a model with the smallest loss or highest accuracy?

I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data): Model A: Accuracy: ~91%, Loss: ~0.01 Model B: Accuracy: ~83%, Loss: ~0.003 The ...
1
vote
0answers
23 views

Using LSTM model to train spatial inputs

I have an $x$-$y$ plane, inside that plane I have 9 paths $(p_1, p_2, \dots, p_3)$. Each path is classified into one of the three classes $(c_1, c_2, c_3)$. Each path has 100 coordinates points i.e $((...
0
votes
1answer
156 views

Feeding YOLOv4 image data into LSTM layer?

How would one extract the feature vector from a given input image using YOLOv4 and pass that data into an LSTM to generate captions for the image? I am trying to make an image captioning software in ...
2
votes
2answers
201 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
0
votes
1answer
49 views

Is the working of RNNs, LSTM and GRU sequential or parallel?

You take any blog or any example and all they tell you about is the given picture below. It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in ...
1
vote
1answer
652 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
0
votes
0answers
13 views

Is it possible and if so does it make sense to have dense layers in between LSTM layers?

I am new to LSTMs and I was wondering if it is possible to have LSTM layer then dense then LSTM again and does it make sense?
13
votes
3answers
8k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
3
votes
1answer
138 views

What's the difference between LSTM and GRU?

I have been reading about LSTMs and GRUs, which are recurrent neural networks (RNNs). The difference between the two is the number and specific type of gates that they have. The GRU has an update gate,...
1
vote
1answer
2k views

What’s the difference between LSTM and RNN?

What's the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
4
votes
2answers
4k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
0
votes
0answers
15 views

Text generation with LSTM with multiple correlated inputs

I am currently working on a music-generation project, inspired by an already existing project called Deepbach. My dataset are the Bach chorales, which are all composed of 4 independent (but related) ...
9
votes
1answer
214 views

How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM? So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), ...
0
votes
0answers
23 views

During Backpropagation in LSTM, why is the previous output $h_{t-1}$ considered constant w.r.t any $W$ while computing derivative?

I've just started learning LSTM, and some points in the process of calculating the gradients are getting me confused. Say, for example, we want to compute $\frac{\partial}{\partial W_i}L$, where $L$ ...
11
votes
1answer
3k views

What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, ...
0
votes
0answers
80 views

Understanding Stable Baselines Custom Policies

I'm trying to understand the structure of the custom recurrent policy introduced in the documentation of the Stable Baselines: How exactly is the Lstm NN constructed? (check code below) From what I ...
0
votes
0answers
9 views

Is the information in the hiddenstate of a RNN worth processing further after the input passes the RNN?

I hope the question is understandable. I just wanted to ask if the hidden state, which is passed through the timesteps/cells of an RNN/LSTM/GRU to deliver information from $\text{cell}_{i-1}$ to the $\...
2
votes
1answer
36 views

Understanding LSTM through example

I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states. Lets assume that ...
0
votes
0answers
19 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
0
votes
0answers
61 views

Time series prediction using LSTM and CNN-LSTM: which is better?

I am working on LSTM and CNN to solve the time series prediction problem. I have seen some tutorial examples of time series prediction using CNN-LSTM. But I don't know if it is better than what I ...
2
votes
0answers
53 views

Is there a mechanism in the human brain that works analog to LSTMs?

Is there a mechanism in the human brain that works analog to LSTMs? Is there a biological/neuroscientific interpretation of LSTMs and recurrent neural networks? How do long-term and short-term ...
2
votes
0answers
37 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
2
votes
1answer
28 views

Does this diagram represent several LSTMs, or one through several timesteps?

I'm trying to read this paper describing Google's LSTM architecture for machine translation. It features this diagram on page 4: I'm interested in the encoder block, on the left. Apparently, the pink ...
1
vote
0answers
30 views

How does Google's 2016 GNMT architecture work?

I'm trying to read this paper describing Google's LSTM architecture for machine translation from 2016. However, I'm getting stuck as certain things are described too vaguely for me. This is a picture ...
1
vote
0answers
46 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
1
vote
1answer
234 views

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data. I ...
2
votes
1answer
80 views

How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
0
votes
0answers
18 views

How to afine the extremity values in regression prediction with Keras?

I made a stack of bidirectional LSTM layers following by Dense layers (with swish activation functions) in order to predict a continuous value between 0 and 2. I compiled the model with ...
3
votes
1answer
60 views

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information? What ...
1
vote
1answer
23 views

How do LSTMs work if the following two matrices are not able to be multiplied?

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance: The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$ The input data at timestep t ...
3
votes
0answers
91 views

What sort of Neural Network is best suited to predicting a future purchase?

I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4. Now I'm trying to do a NN that can make a prediction. ...
0
votes
0answers
28 views

In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?

There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs. Both of these sources mention the reason LSTMs are ...
2
votes
1answer
62 views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
1
vote
2answers
409 views

Why are RNNs better than MLPs at predicting time series data?

Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data. What I don't understand is why RNNs are also ...
0
votes
0answers
22 views

Stateful many-to-many RNN generating artefacts at regular intervals

I am training a stateful LSTM network, with a time series consisting of about 500000 data points spread over 5 years. This time series is split up to batches of 100 timesteps, and fed into the network....
8
votes
1answer
2k views

Will attention based networks prevail over RNN and LSTM? [closed]

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
10
votes
1answer
1k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
1
vote
0answers
23 views

Duplicating calculations in CNN-LSTM architecture

I want to use frames from video game and analyze them using CNN and LSTM. But when I have the model defined like that ...
4
votes
0answers
74 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
1
vote
1answer
28 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
0
votes
0answers
15 views

Is there a model suitable to predict one correct value based on a 2D input series?

I am using an encoder-decoder architecture, with 2 layers each in the encoder and decoder and 128 neurons in each hidden layer. The inputs are in a 2D form: one column has the days and the other ...
0
votes
1answer
51 views

What type of model should I fit to increase accuracy?

Currently, I'm working on 6-axis IMU(Inertial Measurment Unit) dataset. This dataset contain 6 axis IMU data of 7 different drivers. The Imu sensor attached on vehicle. The drivers drives same path. ...
0
votes
0answers
19 views

Avoiding Overfitting with a large LSTM net on a small amount of data

1. Context I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. There'...
0
votes
0answers
13 views

Python code in LSTM to look at selective history

My dataset has 3 columns - a,b,c. Using b (and its history), I wish to predict c. Using list function and converting to array, I can tell python to look at last 20 b's for any b, as input to predict c....