Questions tagged [long-short-term-memory]
For questions related to the long-short term memory (LSTM), which refers to a recurrent neural network architecture that uses LSTM units. The first LSTM unit was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber in the paper "Long-Short Term Memory".
291
questions
0
votes
1
answer
32
views
When are Transformers better than LSTMs in time-series tasks such as classification?
I’m working on a time-series classification problem and trying to decide whether to use a Transformer or an LSTM.
From what I’ve learned, Transformers are better suited for capturing long-range ...
1
vote
1
answer
18
views
Can I do incremental learning with different loss function in neural networks?
I have a saved tensorflow neural network model. I was wondering if it's possible to incrementally train the model but with different nt loss function.
- 111
0
votes
0
answers
12
views
Improving forecast for LSTM with additional data
There are two timeseries X and Y. The timeseries X spans the duration [1 Jan, 20 Jan ] and the Y spans [1 Jan, 25 Jan]. I am interested in timeseries forecasting of variable X for the duration [21 ...
- 101
0
votes
0
answers
25
views
Simultaneous forecasting and classification
I'm working on a project where I need to perform both forecasting (regression) and classification using time series data. The dataset is labelled.
I've been exploring LSTM networks due to their ...
- 1
0
votes
0
answers
13
views
How can a RNN with 256 cells accept a input of any size?
I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model.
...
0
votes
0
answers
13
views
How is the accuracy as a metric in a Keras machine learning model calculated? Is it a valuable metric for LSTM
I'm training a LSTM neural network for time series prediction in Keras. During the training of the model, the loss (mse) gradually decreases each epoch, but the accuracy as well as the validation ...
- 1
0
votes
0
answers
12
views
Possible Reasons for the Discrepancy in Trainable Parameters of the Extended DeepConvLSTM Model
I have implemented DeepConvLSTM baseline Model input are 60×d frames each representing 60 samples with d features. Frames are fed into four consecutive convolution layers with standard rectified ...
- 41
1
vote
1
answer
23
views
LSTM with multiple data streams
I am working on the following problem:
I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
- 11
0
votes
1
answer
22
views
Sequential models and distribution shift in RL
We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
0
votes
0
answers
21
views
ROC curve for multiclassification - results sound not correct
I'm working on a multiclassification task using LSTM algorithm, i generated my roc curve plots but they give scores like 1 , 0.99, 0.97 however i have an accuracy of 0.97, Precision 0.65, Sensitivity/...
- 1
0
votes
0
answers
41
views
How to setup correctly a sequence generation task with RL/policy gradient learning?
I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points.
So, I have the following model and loss function:
...
- 1
0
votes
0
answers
41
views
Updating custom output layers of an LSTM network
I have a text generation task learning to predict the next word with an LSTM network with multiple output layers.
After the generation of a sentence has finished, I calculate a reward for the whole ...
- 1
0
votes
1
answer
21
views
Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]
I am new to Seq2Seq and hope to find a proper guildances, advices.
I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github
I want to ask ...
- 3
0
votes
0
answers
19
views
How do I find a similar RNN as a starting point?
I am new to machine learning and neural networks and I want to create a neural network for a study project.
I would like to create a RNN, that uses one (A) or several time series (with the same length,...
- 1
0
votes
0
answers
47
views
What are the *non-cost-related* reasons RNN+Attention underperform Transformers?
There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
- 146
1
vote
1
answer
40
views
How does the linear layer step work and what should I do at the end of the LSTM?
So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
- 111
1
vote
0
answers
18
views
good period prediction but bad magnitude using keras LSTM neural network
I want to predict the voltage of a battery along time using neural networks. This voltage is read using an ADC and generates a charge/discharge profile that ideally looks like this:
Which goes very ...
- 111
0
votes
1
answer
45
views
In theory, LSTM should be adequate to solve my problem, but it doesn't
I am currently doing my Master's Thesis in the area of wireless sensing using radio waves. I have 4 receiver distributed in the corners of my room and each of them records so called Channel Impulse ...
- 111
0
votes
0
answers
30
views
LSTM Ensemble: Combine low, mid, and high frequency time series data
I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
- 121
0
votes
2
answers
277
views
LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.
My ...
- 3
0
votes
0
answers
11
views
Can I combined the trained model between different source but same model structure?
Here I got two different deep-learning models that were trained by LSTM and time-series data.
The data is the usage percentage of CPU from two different computers.
Each computer job was the same. It ...
- 103
0
votes
0
answers
39
views
Not pre-trained binary transformer model
I stacked with a problem. My default Transformer model totally does not learn how to evaluate python logical expressions, like: '(False and not True) xor False or (not False and False)'. Model should ...
0
votes
1
answer
619
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would ...
- 103
0
votes
0
answers
34
views
Is this lstm diagram correct?
I made an LSTM diagram, but do not know if it is correct. Can you point out any errors in case there are any?
- 101
0
votes
0
answers
17
views
Tips for getting LSTM to train for next word predcition
I am trying to train an LSTM network for next work prediction. I have scraped a rather large dataset from Wikipedia of country descriptions. I have done normal preprocessing (removing punctuation and ...
- 101
1
vote
0
answers
41
views
The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination
I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model:
This model first archived about 0.03 accuracy score and gradually ...
- 11
0
votes
1
answer
24
views
Regression Model diverging after adding a new feature with higher variance and magnitude
In a time series regression problem I'm predicting "change" rather than the actual intended value i.e
Instead of:
...
1
vote
1
answer
53
views
Transfer Learning for Solar Energy Production Forecasting with LSTM: Generalized vs. Specialized Models
I am working on a solar energy production forecasting problem using LSTM multi-step models to predict 1/4/8h ahead of solar energy production for different solar installations. Our goal is to help ...
1
vote
1
answer
69
views
Recognize patterns within random sequences
I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
- 111
0
votes
1
answer
585
views
Difference between sequence length and hidden size in LSTM
It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs.
For example, in the next ...
- 31
3
votes
2
answers
1k
views
Can LSTM model use ReLU or LeakyReLU as the activation funtion?
Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
- 67
1
vote
1
answer
121
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 11
1
vote
1
answer
35
views
Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?
I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
- 111
0
votes
1
answer
90
views
Why final memory state equals to the last hidden state of entire hidden state sequence?
when return_sequences=True and return_state=True, a TensorFlow LSTM outputs the hidden states of the LSTM cell along with the ...
- 123
1
vote
0
answers
20
views
SCINet: how does interactive learning work?
i'm having some trouble understanding how does the basic building block of a SCINet works. In the paper the author describes the SCI-block with the following figure:
In which $\phi$, $\theta$, $\eta$ ...
0
votes
0
answers
26
views
How can I transform the LSTM output to an embedding matrix of actions?
Page 3 of the paper Feudal Networks for Hierarchical Reinforcement Learning describes producing an 'embedding matrix' $U$ of size $(|a| \times k)$ from the output of an LSTM, given a $(d \times 1)$ ...
- 1
0
votes
0
answers
17
views
Text Classification Model unable to learn
I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases.
I have researched for possible ...
- 1
0
votes
1
answer
63
views
How to identify overfitting in LSTM-RNN using metrics?
How can I identify overfitting on a RNN-LSTM with the following metrics: RMSE, MSE, RAE, R-squared ? I have searched papers and google results. I don't see something clear to my mind. Also I rarely ...
- 105
0
votes
0
answers
32
views
Modeling the previous inputs to affect next output in Machine learning
I am working on a dataset contains one output variable and a number of input variables.The data looks like the following:
Y, X1, X2, X3, X4
7, 5, 0.7, 8, 9
3, 6, 0.3, 9, 9
....
Where Y is the output ...
1
vote
0
answers
28
views
bad prediction when having noise on the data: LSTM time-series regression
I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
- 11
1
vote
0
answers
14
views
Where exactly is permutation happening in equation 5 of the paper "Learning with Sets in Multiple Instance Regression Applied to Remote Sensing"?
I am reading the article Learning with Sets in Multiple Instance Regression Applied to Remote Sensing about creating an embedding which is order-invariant to inputs ($m_{l}$). They referred to order-...
- 43
2
votes
2
answers
2k
views
Val loss doesn’t decrease after a certain number of epochs
I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss ...
- 65
0
votes
1
answer
141
views
Extremely good results in RNN-LSTM python code!! How can this happen?
I am using this code here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ and more specifically the python code under the (1st) paragraph "...
- 105
1
vote
2
answers
114
views
How does Seq2Seq with attention actually use the attention (i.e. the context vector)?
For neural machine translation, there's this model "Seq2Seq with attention", also known as the "Bahdanau architecture" (a good image can be found on this page), where instead of ...
- 181
0
votes
1
answer
99
views
Does sequence matter in LSTM?
In general, as long as the items are in order as per time sequence, does it matter if it's in ascending time sequence or descending time sequence for LSTM within the input vector?
eg:
ABCDEF -> G
...
0
votes
1
answer
112
views
Choice of LSTM for price prediction
I have a dataset with features (f) for different stocks (S) and want to infer for price using an LSTM model. Here is my df:
year
S1_price
S1_f1
S1_f2
S2_price
S2_f1
S2_f2
Sn_price
Sn_f1
Sn_f2
2010
...
- 21
0
votes
1
answer
32
views
What is the training accuracy of this model?
I’m trying to classifiy ECG signals using LSTM and MATLAB, the above plot shows that the training accuracy of the system is 100% but when I apply this code to calculate and get the accuracy I get only ...
0
votes
1
answer
80
views
Limitations of LSTMs
I'm training an LSTM model for classification on accelerometer data, and I get better results when I downsample the signal to 25 Hz than when I use a 50 Hz signal.
I use the same time frame of 1.5 ...
- 121
0
votes
0
answers
94
views
How is it possible to use batches of data from within the same sequence with an LSTM?
ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless?
Using the example of an LSTM, it has a hidden ...
- 1,346
2
votes
2
answers
124
views
LSTM exploding? - multiple parallel time series with multiple variables
I have the following situation:
Stock
Time_Stamps
Feature_1
Feature_2
Feature_n
Price
Stock_1
2019
0.5
1.0
1.0
100
Stock_1
2020
0.7
1.3
0.9
90
Stock_2
2019
0.3
0.9
1.1
110
Stock_2
2020
0.2
0.8
1....
- 21