Questions tagged [long-short-term-memory]
For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".
307
questions
1
vote
0
answers
25
views
Music simplifying modeling
I'm working on a machine learning task involving a unique dataset of piano tracks, represented as arrays with the shape (200, 10, 5000). Here, 200 is the number of tracks, 10 represents the number of ...
- 111
0
votes
0
answers
22
views
What is currently the best options for a non sequence-parallel RNN model?
It's well known that sequence-parallel models (like Transformer, Mamba, RWKV etc.) are by design unable to efficiently solve inherently sequential problems and even much simpler state tracking tasks (...
0
votes
0
answers
22
views
Why can't I replicate the validation loss from a Keras tuner (LSTM)
I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the ...
- 101
0
votes
0
answers
13
views
Spikes in Loss During Training both train/val datasets with LSTM
I'm seeing good results I think, but I want to understand why these spikes in loss are occuring.
As you can see, it would appear that my training is working as it should, but every 200 or so epochs ...
- 101
0
votes
0
answers
12
views
What is the reason for the difference between the expected input tensor order for LSTM and Conv1d?
What is the reason for the difference between the expected input tensor order for LSTM and Conv1d?
Say I have an input tensor for time series data of shape ...
- 1
0
votes
0
answers
20
views
Architecture of an LSTM with multiple (dependent?) time series
If you have multiple time series data for a given problem (e.g predicting house prices and data is available per city).
Per city there is a list of features and the target feature.
If you want to ...
- 1
0
votes
0
answers
13
views
Why time based neuron still needed when the time-series data can be converted to time-freq domain (image) and use CNN for that?
LSTM, BI-LSTM, GRU, RNN are time-step based neuron. Why it still needed specifically for time-series data?
I mean, we can just transform the time-series data into spectrogram and use CNN for that.
For ...
0
votes
0
answers
19
views
Formatting time-series data for LSTM model
To my knowledge, LSTM models take 3D input data of the format (batch_size, timesteps, features). But, when creating the batches, I've often seen them literally add a batch where the data is just ...
1
vote
1
answer
27
views
How do I prepare a multi group time series dataset into a supervised learning one?
ML newbie here, I have a time series dataset that looks like this:
...
- 11
1
vote
1
answer
46
views
What is the exact purpose of input modulation gate in LSTMs?
Basically, I was learning about LSTMs where I found LSTMs are made up of three gates: The forget gate, input gate and output gate. However, I came across some sources that state there is a fourth gate ...
- 11
0
votes
0
answers
24
views
Distribution function for lstm and MLP
I have total of 6300 samples, 5800 of which are training data, and 500 of which are testing data. We compare the performance of LSTM and multilayer perceptron (MLP) with one hidden layer in terms of ...
- 101
0
votes
1
answer
22
views
Spatial vs spatiotemporal methods for object counting in low frame-rate videos
I'm currently working on an object counting/density estimation task using low frame rate video (~2 fps) in a traffic setting. I've explored a lot of literature on both spatial methods (i.e. using only ...
- 11
0
votes
0
answers
74
views
Any reasons LSTM does not pick up any patterns?
I'm trying to teach an LSTM to predict the next values in 3 related series. (Financial data)
Unfortunately, it looks like I made some basic mistake and this network never gets past just returning ...
- 111
0
votes
0
answers
27
views
Preparation of multivariate time series data
I am doing a university project on index/stock price prediction.
I plan to use a combined cnn-lstm model, and I have several different types of data: Open High Low Close Volume, values, fundamental ...
- 1
0
votes
0
answers
13
views
Is it possible to apply transfer learning between Temporal Fusion Transformer and sequential architecture LSTM and GRU
If TFT is a pretrained model, is it possible to transfer the weights to sequential neural network models like LSTM,BILSTM and GRU.
0
votes
1
answer
108
views
xLSTM parallel computation - mismatch in dimensions
In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
- 101
0
votes
1
answer
33
views
Can I scale subsets of my dataset independently to handle different feature ranges?
I am currently making an LSTM model to predict the change in value of a stock for each day. My dataset consists of 4 years of data for 30 different stocks along with some financial metrics such as RSI,...
0
votes
0
answers
66
views
Drum sound classification using RNN issues - help needed
I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound ...
- 1
1
vote
2
answers
122
views
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at ...
0
votes
0
answers
36
views
What is the input size and sequence length for lstm when the input is audio data where each audio file is sampled at the rate of 3000hz?
I have music data which contains raw vocals of some indian classical and devotional songs and I have segmented each file into 20 seconds files. I have used librosa.load function with a sample rate of ...
0
votes
0
answers
32
views
"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book
I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote:
Neural networks excel ...
- 101
1
vote
0
answers
286
views
Glass Degradation Video Prediction
We are working on the subject above, where a sequence of $n$ glass frames forms an example with an associate target that is a video of the glass future state.
We would like to know if there exists an ...
0
votes
0
answers
36
views
How can I process financial stock data before passing it to an LSTM for time series classification?
I'm trying to make an LSTM that can classify the next day of a stock as either 1 or 0 for going up or down. The issue I've been having is that Keras tuner seems to stay at a constant value of val_loss ...
0
votes
0
answers
18
views
Extracting features from multiple curves
I am building a model that predicts the SOH of a lithium ion battery.
My data are from 600 battery charge cycles as follows: for each cycle I have 3 curves each of length 128 samples: voltage, current ...
0
votes
1
answer
95
views
How LSTM really decide what to forget and not?
Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" ...
- 3
0
votes
0
answers
53
views
Realtime Inference of Stateful Variable-Length Time Series LSTM - Getting around Sequence Length vs Inference Frequency Tradeoff
I am trying to train a neural net for low-latency signal filtering; I've been recommended to use a stateful LSTM architecture for this task, however, having corrected some , it seems to me that trying ...
- 101
1
vote
1
answer
2k
views
When are Transformers better than LSTMs in time-series tasks such as classification?
I’m working on a time-series classification problem and trying to decide whether to use a Transformer or an LSTM.
From what I’ve learned, Transformers are better suited for capturing long-range ...
- 11
1
vote
1
answer
35
views
Can I do incremental learning with different loss function in neural networks?
I have a saved tensorflow neural network model. I was wondering if it's possible to incrementally train the model but with different nt loss function.
- 111
0
votes
0
answers
13
views
Improving forecast for LSTM with additional data
There are two timeseries X and Y. The timeseries X spans the duration [1 Jan, 20 Jan ] and the Y spans [1 Jan, 25 Jan]. I am interested in timeseries forecasting of variable X for the duration [21 ...
- 101
0
votes
0
answers
28
views
Simultaneous forecasting and classification
I'm working on a project where I need to perform both forecasting (regression) and classification using time series data. The dataset is labelled.
I've been exploring LSTM networks due to their ...
- 1
0
votes
0
answers
17
views
How can a RNN with 256 cells accept a input of any size?
I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model.
...
0
votes
0
answers
21
views
How is the accuracy as a metric in a Keras machine learning model calculated? Is it a valuable metric for LSTM
I'm training a LSTM neural network for time series prediction in Keras. During the training of the model, the loss (mse) gradually decreases each epoch, but the accuracy as well as the validation ...
- 1
0
votes
0
answers
12
views
Possible Reasons for the Discrepancy in Trainable Parameters of the Extended DeepConvLSTM Model
I have implemented DeepConvLSTM baseline Model input are 60×d frames each representing 60 samples with d features. Frames are fed into four consecutive convolution layers with standard rectified ...
- 41
1
vote
1
answer
48
views
LSTM with multiple data streams
I am working on the following problem:
I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
- 11
0
votes
1
answer
44
views
Sequential models and distribution shift in RL
We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
0
votes
0
answers
23
views
ROC curve for multiclassification - results sound not correct
I'm working on a multiclassification task using LSTM algorithm, i generated my roc curve plots but they give scores like 1 , 0.99, 0.97 however i have an accuracy of 0.97, Precision 0.65, Sensitivity/...
- 1
0
votes
0
answers
43
views
Updating custom output layers of an LSTM network
I have a text generation task learning to predict the next word with an LSTM network with multiple output layers.
After the generation of a sentence has finished, I calculate a reward for the whole ...
- 1
0
votes
1
answer
29
views
Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]
I am new to Seq2Seq and hope to find a proper guildances, advices.
I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github
I want to ask ...
- 3
0
votes
0
answers
21
views
How do I find a similar RNN as a starting point?
I am new to machine learning and neural networks and I want to create a neural network for a study project.
I would like to create a RNN, that uses one (A) or several time series (with the same length,...
- 1
1
vote
1
answer
139
views
How does the linear layer step work and what should I do at the end of the LSTM? [closed]
So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
- 111
0
votes
1
answer
70
views
In theory, LSTM should be adequate to solve my problem, but it doesn't
I am currently doing my Master's Thesis in the area of wireless sensing using radio waves. I have 4 receiver distributed in the corners of my room and each of them records so called Channel Impulse ...
- 111
0
votes
0
answers
112
views
LSTM Ensemble: Combine low, mid, and high frequency time series data
I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
- 121
0
votes
2
answers
788
views
LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.
My ...
- 3
0
votes
1
answer
2k
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would ...
- 103
1
vote
0
answers
48
views
The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination
I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model:
This model first archived about 0.03 accuracy score and gradually ...
- 11
0
votes
1
answer
26
views
Regression Model diverging after adding a new feature with higher variance and magnitude
In a time series regression problem I'm predicting "change" rather than the actual intended value i.e
Instead of:
...
1
vote
1
answer
83
views
Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models
I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help ...
1
vote
1
answer
235
views
Recognize patterns within random sequences
I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
- 111
0
votes
1
answer
2k
views
Difference between sequence length and hidden size in LSTM
It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs.
For example, in the next ...
- 31
3
votes
2
answers
2k
views
Can LSTM model use ReLU or LeakyReLU as the activation funtion?
Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
- 67