Questions tagged [long-short-term-memory]

For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".

Filter by
Sorted by
Tagged with
2
votes
1answer
101 views

Do I need LSTM units everywhere in the network?

I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and ...
2
votes
1answer
336 views

Why can we approximate the joint probability distribution using the output vector of an LSTM?

In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that \begin{equation} P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1}) \end{equation} They also state that, in the LSTM ...
10
votes
3answers
2k views

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
4
votes
1answer
172 views

How should I design the LSTM architecture for multivariate time series forecasting problems?

There is plenty of literature describing LSTMs in a lot of detail and how to use them for multi-variate or uni-variate forecasting problems. What I couldn't find though, is any papers or discussions ...
1
vote
2answers
470 views

Why are RNNs better than MLPs at predicting time series data?

Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data. What I don't understand is why RNNs are also ...
4
votes
1answer
579 views

Why do we need multiple LSTM units in a layer?

What is the point of having multiple LSTM units in a single layer? Surely if we have a single unit it should be able to capture (remember) all the data anyway and using more units in the same layer ...
2
votes
1answer
36 views

Modifying LSTM to include forecast

I am looking at LSTM example here However, I am not sure how to modify the setup if I have forecast available (assuming perfect forecast) for TEMP: Temperature and PRES: Pressure at time t. i.e. <...
4
votes
1answer
161 views

Object IN/OUT counting using CNN+RNN

I am building a video analytics program for counting moving things in a video. I am detecting bicycles and nothing else. I run object detection using the SSD mobile-net model in all the frames and ...
1
vote
1answer
59 views

Is there an LSTM-based unsupervised learning algorithm to label a dataset of curves?

I have a big amount of light curves (image below). I am trying to label the points as signal or background (the signal appears usually periodically, several times, for a given light curve). More ...
2
votes
0answers
177 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
1
vote
0answers
17 views

Training LSTM with class imbalance

I need to train an LSTM on some light curves, in order to find a signal (there are 2 classes signal and background). However the signal (data points corresponding to signal) is around 100 times less ...
3
votes
0answers
193 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
2
votes
0answers
43 views

RNN: Different test results on balanced and unbalanced data

I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each ...
4
votes
0answers
86 views

How do the relative number of cells between neighboring stacked LSTM layers affect the network's behavior?

It seems that stacking LSTM layers can be beneficial for some problem settings in order to learn higher levels of abstraction of temporal relationships in the data. There is already some discussion on ...
1
vote
1answer
96 views

DQN Q-values are static

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
2
votes
0answers
44 views

Multi-field text input for LSTM

I'm using LSTM to categorize medium-sized pieces of text. Each item to be categorized has several free-form text fields, in addition to several categorical fields. What is the best approach to using ...
1
vote
0answers
74 views

Why validation performance is unstable for my LSTM based model (labelling problems)?

I have trained a recurrent neural network based on 1 stack of LSTM cells. I use it to solve a classification problem. The RNN cell has 48 hidden states. The output of the last unfolded LSTM cell is ...
1
vote
1answer
419 views

LSTM is not converging

I am writing my first LSTM network and I would really appreciate if someone can tell me if it is right (the loss seems to go down very slowly and before playing around with hyper parameters I want to ...
1
vote
0answers
58 views

Predicting sine using LSTM: Small output range and delayed output?

I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise. I have tweaked the model, changing the output layer ...
1
vote
0answers
22 views

How to train chat bot on infinite non-stationary data?

I have continual simulated data of million sentences of two simulated persons talking to each other in a room and I want to model one of the persons speech. Now, during this period things in the room ...
1
vote
0answers
479 views

Experiment shows that LSTM does worse than Random Forest... Why?

LSTM is supposed to be the right tool to capture path-dependency in time-series data. I decided to run a simple experiment (simulation) to assess the extent to which LSTM is better able to understand ...
9
votes
3answers
2k views

Should I choose a model with the smallest loss or highest accuracy?

I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data): Model A: Accuracy: ~91%, Loss: ~0.01 Model B: Accuracy: ~83%, Loss: ~0.003 The ...
2
votes
1answer
89 views

Is my Neural Network program fully connected?

I have the following program for my neural network: ...
2
votes
0answers
41 views

How does a neural network output text box location data?

I'm interested in creating a convolutional neural network or LSTM to locate text in an image. I don't want to OCR the text yet, just find the text regions. Yes, I know Tesseract and other systems can ...
1
vote
0answers
31 views

Kera's (normal) LSTM uses the GPU?

I'm running Kera's LSTM (not CuDNNLSTM) but I notice my GPU is under load. I need recurrent dropout, so I can only stick with <...
1
vote
0answers
38 views

how to estimate which item is bought most using AI

Problem: Maintaining the products is a big task for a retailer. If we can estimate using AI to predict which products will sell the most, we can maintain sufficient stocks of the product without ...
1
vote
1answer
597 views

LSTM language model not working

I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons ...
1
vote
0answers
573 views

Number of nodes in hidden layer for LSTM

I am starting to learn LSTM by understanding how it is used for creating a char-RNN and had a fundamental question. Does the number of nodes in the hidden layer need to be the same as that of the ...
1
vote
1answer
42 views

Is there bidirection sequence-to-sequence neural machine translation?

I have heard about bidirectional RNN LSTM units (endcoders-decoders), but my question is - is there bidirectional neural machine translation, that uses A->B weights for the translation in the opposite ...
2
votes
0answers
57 views

Is there a mechanism in the human brain that works analog to LSTMs?

Is there a mechanism in the human brain that works analog to LSTMs? Is there a biological/neuroscientific interpretation of LSTMs and recurrent neural networks? How do long-term and short-term ...
0
votes
1answer
26 views

Reduce the effect of excessive zeros

I am working on an autoregression problem where I use sequential LSTM. My target is well defined, but I think I am facing a problem with the features. As the features were non-stationary, then I ...
1
vote
1answer
243 views

Are 1D CNNs really the appropriate model for human activity recognition?

This article on human activity recognition states that 1D convolutional neural networks work the best on the classification of human activities using data from the accelerometer. But I think that ...
1
vote
0answers
56 views

How could I learn tree paths given word embeddings?

I need to map from a vector space representation onto a tree structure. A possible solution: given a word vector as input, produce a path in the tree from the root down to the node that most closely ...
2
votes
1answer
113 views

Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well ...
3
votes
2answers
173 views

Price Movement Forecasting Issue

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
3
votes
2answers
57 views

Problem extracting features from convolutional layer where the dimensions are big for feature maps

I have trained a convolutional neural network on images to detect emotions. Now I need to use the same network to extract features from the images and use them to train an LSTM. The problem is: the ...
1
vote
0answers
26 views

If there are several computers on a subnet, can training time be reduced by distributing the work across them?

We have multiple computers and the ability to ssh between them. What are options using either Java, C/C++, JavaScript, or Python to distribute our learning tasks? We will be using DCNN, DQN, and LSTM ...
1
vote
1answer
95 views

How to understand marginal loglikelihood objective function as loss function (explanation of an article)?

I am reading article https://allenai.org/paper-appendix/emnlp2017-wt/ http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf about training neural network and the loss function is mentioned ...
3
votes
2answers
619 views

What should I do when I have a variable-length sequence when instantiating an LSTM in Keras?

In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that ...
2
votes
1answer
121 views

Difficulty understanding Keras LSTM fitting data

I'm try to train a RNN with a chunk of audio data, where X and Y are two audio channels loaded into numpy arrays. The objective is to experiment with different NN designs to train them to transform ...
2
votes
0answers
105 views

Structure of a multilayered LSTM neural network?

I implemented a LSTM neural network in Pytorch. It worked but I want to know if it worked the way I guessed how it worked. Say there's a 2-layer LSTM network with 10 units in each layer. The inputs ...
13
votes
4answers
10k views

Where can I find the original paper that introduced RNNs?

I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?
1
vote
1answer
69 views

Why would giving my AI more data make it perform worse?

So I trained an AI to generate shakespeare, which it did somewhat well. I used this 10,000 character sample. Next I tried to get it to generate limericks using these 100,000 limericks. It generated ...
1
vote
2answers
168 views

Are artificial networks based on the perceptron design inherently limiting?

At the time when the basic building blocks of machine learning (the perceptron layer and the convolution kernel) were invented, the model of the neuron in the brain taught at the university level was ...
1
vote
1answer
105 views

How do the current input and the output of the previous time step get combined in an LSTM?

I am currently looking into LSTMs. I found this nice blog post, which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers. The input $X_t$,...
1
vote
2answers
77 views

Need Help With LSTM Neural Networks

I have been researching LSTM neural networks. I have seen this diagram a lot and I have few questions about it. Firstly, is this diagram used for most LSTM neural networks? Secondly, if it is, ...
15
votes
1answer
14k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
3
votes
1answer
53 views

Using different timesteps for features and target value

I would like to know whether it's wrong; when working with time series data; to use daily prices as features and the price after 3 days as target. Is this correct or should I use the next-day price ...
8
votes
1answer
2k views

Will attention based networks prevail over RNN and LSTM? [closed]

There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
4
votes
1answer
241 views

What would be the best approach to teach an AI to learn how to "sing" along a beat?

I have heard and read about HyperGAN, LSTM and a few other techniques, but I have a hard time piecing the overall concept together. End Goal Being able to input an instrumental and get an output of ...