Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
3
votes
0answers
34 views

RNN models displays upper limit on predictions

I have trained a RNN, GRU, and LSTM on the same dataset, and looking at their respective predictions I have observed, that they all display an upper limit on the value they can predict. I have ...
3
votes
0answers
42 views

Why don't the neural networks inside LSTM cells contain hidden layers?

I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers. Why don't LSTM cells have more complex neural ...
4
votes
0answers
23 views

Training an RNN to answer simple quesitons

I would like to train an RNN to follow the sentences: "Would you like some cheese"? with "Yes, I would like some cheese." So whenever the template "Would you like some ____?" appears then RNN ...
3
votes
0answers
27 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
3
votes
1answer
47 views

Can dropout layers not influence LSTM training?

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows: ...
3
votes
2answers
46 views

How to use LSTM to generate a paragraph

A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the ...
0
votes
0answers
24 views

Outliers detection problem in neural networks

Assuming we have big m x n input dataset with m x 1 output vector. It's a classification problem with only two possible values: either 1 or 0. Now the problem is that almost all elements of the output ...
8
votes
1answer
127 views

How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM? So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), ...
3
votes
1answer
53 views

Why can't LSTMs keep track of the “important parts” of a sequence?

I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format. For ...
2
votes
0answers
20 views

How are batch statistics computed in Recurrent Batch Normalization?

I'm implementing recurrent BN per this paper in Keras, but looking at it and those citing it, a detail remains unclear to me: how are batch statistics computed? Authors omit explicit clarification, ...
1
vote
0answers
26 views

What is the correct input shape for my LSTM network?

My professor gave us a workshop where we have to do classification of a dataset of ECG signals between healthy and unhealthy types using LSTM. Each signal consists of 1,285 time steps. What my prof ...
3
votes
1answer
28 views

Can non-sequential deep learning models outperform sequential models in time series forecasting?

Can a CNN (or other non-sequential deep learning models) outperform LSTM (or other sequential models) in time series data? I know this question is not very specific, but I experienced this when ...
3
votes
0answers
72 views
+50

Why do regression LSTMs learn high to low inputs significantly better than low to high?

The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361. I have ...
1
vote
1answer
54 views

What is the difference between Kaldi and DeepSpeech speech recognition systems in their approach?

I would like to know how do Kaldi and DeepSpeech speech recognition systems differ algorithmically? Which one would be more accurate for continuous speech in time?
2
votes
1answer
16 views

How to represent integer values in sequence to sequence prediction task in encoder-decoder LSTM?

I have a large 2D grid having 30k rows and 35k columns, so a total of 30x35k grid cells. Each grid cell is represented by a unique integer number (identity of grid cell). I have several trajectories ...
1
vote
0answers
43 views

How to train LSTM score prediction with very little data? (Bounty to be added)

I am trying to make a text score prediction network, and my dataset have 500 samples only. I know there is a public dataset called the ASAP Dataset. I have tested my model ...
3
votes
0answers
20 views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
4
votes
2answers
36 views

Using a neural network to identify a stable region within a set of data?

I am working on a problem in which I am attempting to find a stable region in a spiral galaxy. The PI I'm working with asked me to use machine learning as a tool to solve the problem. I have created ...
2
votes
2answers
71 views

Why can't LSTMs tell a long story?

There is a recent trend in people using LSTMs to write novels. I haven’t attempted this myself. From what I’m hearing, they can tell a story, but it seems they lose the context of the story rather ...
8
votes
2answers
653 views

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs ...
2
votes
2answers
57 views

What is hidden state exactly in LSTM and RNN?

I'm working on research rn using LSTM as an encoder decoder in hopes to make inferences. The reason we are using encoder decoder for this is because there is hopes that the hidden state given by the ...
0
votes
0answers
21 views

Why is the value range of my LSTM model's prediction different from my test labels?

I am using LSTM to do time-series anomaly detection. The data is an hourly sensor input across multiple years (i.e. the global_active_energy attribute of the dataset from https://www.kaggle.com/uciml/...
1
vote
2answers
80 views

What are the standard problems for CNNs and LSTMs?

What are the standard (or baseline) problems (or at least common ones) for CNNs and LSTMs? As an example, for a feed-forward neural net, a common problem is the XOR problem. Is there a standard ...
2
votes
0answers
36 views

How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
2
votes
1answer
61 views

Why does an LSTM cycle on initialisation?

I initialised an LSTM with Xavier initialisation, although I've found this occurs for all initialisations I have tested. When initialised, if the LSTM is tested with a random input, it will get stuck ...
4
votes
1answer
115 views

dimensions of hidden layer and cell state layer in LSTM

I was following some examples to get familiar with tensorflow LSTM related api, but noticed that all LSTM initialization functions require only num_units parameter which denotes number of hidden units ...
3
votes
1answer
45 views

What sort of Neural Network is best suited to predicting a future purchase?

I have previously implemented a Neural Network with Back-Propagation that was able to learn Tic-tac-toe and could go pretty well at Connect-4. Now I'm trying to do a NN that can make a prediction. ...
0
votes
2answers
107 views

LSTM network doesn't converge, what should be changed? [closed]

I'm testing out TensorFlow LSTM layer text generation task, not classification task; but something is wrong with my code, it doesn't converge. What changes should be done? Source code: ...
0
votes
0answers
14 views

Make an LSTM model for each class separately

I have a dataset of some activities. The dataset contains the status of different sensors and the label of activity. T trained a model in Keras with the following architecture which models the ...
0
votes
1answer
31 views

How to map X to Y for TensorFlow RNN training data

Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data: X = [[0,0],[0,1],[1,0],[1,1]]; Y = [[0], [1], [1], [0] ]; ...
3
votes
1answer
26 views

Structure discrepancy of an LSTM?

I've found multiple depictions of how an LSTM cell operates. See 2 below: and Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden ...
1
vote
2answers
90 views

What are some examples of LSTM architectures?

I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" ...
0
votes
2answers
37 views

How does an LSTM output the correct dimensions for classes?

Take the below LSTM: input: 5x1 matrix hidden units: 256 output size (aka classes, 1 hot vector): 10x1 matrix It is my understanding that an LSTM of this size ...
1
vote
0answers
27 views

Initial LSTM hidden state and cell

If we use LSTMCell from torch: The initial hidden and cell layers should be CONSTANT (from the first time you run the program) and saved right? Like random seeds? ...
2
votes
0answers
19 views

How can I detect fast and slow motion in videos?

I'm trying to detect if a given video shot is fast or slow motion. Basically, I need to calculate a "video motion" score in a given video sequence, meaning how fast or slow motion the video is. For ...
1
vote
1answer
190 views

LSTM in reinforcement learning

Please tell me that is the LSTM network for the problem of reinforcement learning, as I explain to her what she will get the reward of a prediction, because the output will contain only actions? Well,...
0
votes
1answer
42 views

Preparation of input data

Tell me why my val_acc is always the same and how to solve this problem? I saw several topics on the Internet specifically on this problem but they did not help me (for example, use SGD with different ...
2
votes
0answers
46 views

Spike detection in time series using Artificial Neural Networks

I'm quite new in ANNs. I intend to use ANNs for predicting spike points in time series right before they happen. I've already used LSTM for another scenario, and I know that they can be used in ...
0
votes
1answer
66 views

How to train a LSTM model with multi dimensional data

I am trying to train my model using LTSM layer in Keras (python). I have some problems regarding the data representation and feeding it into the model. My data is 184 XY coodinates encoded as a numpy ...
1
vote
0answers
144 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
1answer
74 views

Would this neural network have short term memory?

I want to design a NN that can remember it's last 7 actions and use them as inputs. So for example it would be able to store words in it's memory. Therefore if it had a choice of 10 different actions, ...
1
vote
1answer
71 views

LSTM text classifier shows unexpected cyclical pattern in loss

I'm training a text classifier in PyTorch and I'm experiencing an unexplainable cyclical pattern in the loss curve. The loss drops drastically at the beginning of each epoch and then starts rising ...
0
votes
0answers
53 views

Why doesnt my lstm model for time series prediction improve after certain level of performance?

I created an lstm model which predicts multioutput sequeances. It takes variable length sequences as input. These sequences are padded with zero to obtain equal length. Note that the time series are ...
0
votes
0answers
23 views

Custom optimizer and word-vector evaluator lstm

I’m using Keras LSTM layers and building a model that is trained off ethics text. I have a problem of often over fitting (the network basically remembers my input corpus as it is very small). I was ...
1
vote
0answers
49 views

Keras CRNN implementation with multiple input images

Hello I am trying to implement a CRNN with multiple input images (in my context it is 6 images) This is a regression problem and output is two real value. And for the CNN block I am thinking of using ...
2
votes
1answer
2k views

Adding BERT embeddings in LSTM embedding layer

I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?
0
votes
0answers
130 views

How Seq2Seq with Bidirectional RNN works?

First of all the scope of the question is as follows - we have Sequence2Sequence architecture with: Decoder: Bidirectional LSTM Encoder: regular (single directional) LSTM What I know: When you ...
1
vote
1answer
85 views

Do I need LSTM units everywhere in the network?

I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and ...
1
vote
1answer
113 views

Why can we approximate the joint probability distribution using the output vector of an LSTM?

In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that \begin{equation} P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1}) \end{equation} They also state that, in the LSTM ...
4
votes
1answer
74 views

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...