I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in sequences and also sequence prediction, so I end up reading about RNN's and more especifically about LSTM.
However, I realized all the examples I read always use a more or less easy predictable dataset like a normal numerical sequence (1,2,3,4..) or a kind of wave form that repeats some specific shape every X samples. In these cases the RNNs can not only fit a good regression on training dataset but also make a reasonably good prediction for the future. On the other side, with more random dataset it mostly can only fit the existing dataset with a good regression but not really predict anything very well due to the random nature of the data, which is understandable.
The question I am then making myself is: what is the best approach to tackle a problem in between those extremes? Say we have a limited weather dataset (limited in the sense of not extensive enough to make good predictions) as an example. Even if the dataset is not complete enough so we can make general good predictions, it is still possible that one can find some specific patterns within the dataset that may happen kind of randomly, namely, that almost everytime we see a sequence of 2-3 days raising the temperature together with humidity and wind speed, it rains (just as a simple example). So even though our model might not learn to make a good day-to-day prediction, it could understand that this kind of pattern happens every once in a while, so whenever it recognizes this small sequence, it can say with some good probability that it will rain. Another example would be analyzing traffic data on a network. You might not be able to predict what is being transmitted but you might be able to say, for example, a constant stream of data (video) is about to be transmitted when you see someone has requested data from youtube.com.
In those scenarios, I'm not looking for a model that generally predicts the next step of sequence, but rather to identify within an "apparent random dataset" when some small patterns take place. For these cases, is the RNN's (like LSTM) still the best recommendation of model? If yes, is there something one might need to adjust or tune in order to achieve that? I also read a bit about transformers and apparently you can give focus to some specific events in comparison with others. Is this maybe a use case for that?
I hope the question is not broad enough and I'm looking forward for some tips and advices with this regards. Thanks
when predicting a probability like suggested by maxy, how should I represent the output vector exactly? Having a practical example:
- I have 3 features (humidity, temperature and hours of sunshine in a day).
- I want to predict the probability of rain on the next day.
So in this case, which I do not want to predict all features, but rather just one of them, how should I design it? I can imagine, from what I understood on the suggestion, I would have something like a LSTM layer, followed probably by a Dense layer and then a Softmax as output layer. To train, I would provide a normalized version of my input (physical values in percentage, degrees and hours) and use a float value between 0.0 and 1.0 as label, where 1.0 is used for the past days that rained and 0.0 for the ones it did not. However this is probably not what was suggested, since I would not be dealing with mean/variance of the temperature, neither with any kind of proability distribution, since my model would then just predict a percentage between 0.0 and 1.0.
Also I am not sure if I follow how to fit step 3 into it. Do you mean using the predicted values to try predicting more steps into the future or as another way of training the model mixing past with predicted data?
Sounds like many questions from my side but I find the approach quite interesting and would like to understand it better to give it a try. But for that, I need to understand how exactly is the data format. If you could use my example above to illustrate what you mean, would be very helpful! Thanks