Recognize patterns within random sequences

Question

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in sequences and also sequence prediction, so I end up reading about RNN's and more especifically about LSTM.

However, I realized all the examples I read always use a more or less easy predictable dataset like a normal numerical sequence (1,2,3,4..) or a kind of wave form that repeats some specific shape every X samples. In these cases the RNNs can not only fit a good regression on training dataset but also make a reasonably good prediction for the future. On the other side, with more random dataset it mostly can only fit the existing dataset with a good regression but not really predict anything very well due to the random nature of the data, which is understandable.

The question I am then making myself is: what is the best approach to tackle a problem in between those extremes? Say we have a limited weather dataset (limited in the sense of not extensive enough to make good predictions) as an example. Even if the dataset is not complete enough so we can make general good predictions, it is still possible that one can find some specific patterns within the dataset that may happen kind of randomly, namely, that almost everytime we see a sequence of 2-3 days raising the temperature together with humidity and wind speed, it rains (just as a simple example). So even though our model might not learn to make a good day-to-day prediction, it could understand that this kind of pattern happens every once in a while, so whenever it recognizes this small sequence, it can say with some good probability that it will rain. Another example would be analyzing traffic data on a network. You might not be able to predict what is being transmitted but you might be able to say, for example, a constant stream of data (video) is about to be transmitted when you see someone has requested data from youtube.com.

In those scenarios, I'm not looking for a model that generally predicts the next step of sequence, but rather to identify within an "apparent random dataset" when some small patterns take place. For these cases, is the RNN's (like LSTM) still the best recommendation of model? If yes, is there something one might need to adjust or tune in order to achieve that? I also read a bit about transformers and apparently you can give focus to some specific events in comparison with others. Is this maybe a use case for that?

I hope the question is not broad enough and I'm looking forward for some tips and advices with this regards. Thanks

UPDATE:

---

when predicting a probability like suggested by maxy, how should I represent the output vector exactly? Having a practical example:

• I have 3 features (humidity, temperature and hours of sunshine in a day).
• I want to predict the probability of rain on the next day.

So in this case, which I do not want to predict all features, but rather just one of them, how should I design it? I can imagine, from what I understood on the suggestion, I would have something like a LSTM layer, followed probably by a Dense layer and then a Softmax as output layer. To train, I would provide a normalized version of my input (physical values in percentage, degrees and hours) and use a float value between 0.0 and 1.0 as label, where 1.0 is used for the past days that rained and 0.0 for the ones it did not. However this is probably not what was suggested, since I would not be dealing with mean/variance of the temperature, neither with any kind of proability distribution, since my model would then just predict a percentage between 0.0 and 1.0.

Also I am not sure if I follow how to fit step 3 into it. Do you mean using the predicted values to try predicting more steps into the future or as another way of training the model mixing past with predicted data?

Sounds like many questions from my side but I find the approach quite interesting and would like to understand it better to give it a try. But for that, I need to understand how exactly is the data format. If you could use my example above to illustrate what you mean, would be very helpful! Thanks

Answer 1

Instead of predicting a single output value (like next temperature), you predict a probability distribution (a range of plausible next temperature values).

Then you can do this:

1. Feed the known past into your model.

   LSTM is a good model choice for this. You could also do something much simpler, e.g. a linear model or a look-up table, just taking the last 5 time-steps as inputs.

2. Predict a distribution over the next inputs.

   For language models, this is a categorical distribution: a probability for every possible next token. (A softmax layer transforms a bunch of numbers into a bunch of probabilities that sum up to one, exactly for this purpose.) For weather data, you could predict the mean and variance of the expected temperature in one hour, same for humidity, etc.

   (Predicting a normal distribution may be too simple because it's unimodal, but you can always predict a mixture of multiple normal distributions instead.)

3. Take a random sample from the distribution that you predicted, add it to your "inputs", and go back to step 1.

You can obviously train your model on historical data (step by step), and the loss function of choice is the cross-entropy loss (which measures the difference between two probability distributions: your output, and a usually very sharp "what really happened" kind-of-distribution).

If your model is flexible enough (e.g. a large LSTM), it will be able to learn (reproduce) the kind of random trends you had in mind. The next question is: how do you make use of the fact that it learned them?

For some known input history, you can repeat the above process to 100 times, and you get 100 possible futures. (Or 100 possible text continuations, in case of a language model.) All of them are somewhat plausible (according to your model).

Often, you want a "representative" future, instead of a random future. You could always pick the most likely prediction (instead of a randomly sampled one). Maybe have a look at "beam search" (for language models). It depends on what you want to do with your model. But the point is, the way you build the model is the same either way.

---

Suggested reading: (all of them "solid theory", but in different directions)

• Kalman Filters
• MacKay's book on Information Theory (free PDF download)
• (Partially Observable) Markov Decision Processes