I have a dataset which contains sequences of event of the following type:
- Timestamp_s
- Timestamp_e
- Event_type
I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count], where each element describes the active events within a specific timeframe (1 for active, 0 for inactive). Note that events are not exclusive, so many events may be active at the same time (even 0), so this isn't a one-hot encoding; it's more like a "many-hot" encoding.
Additionally, I've computed an additional list of [seq_size] sequences, representing the time left until the end of the sequence.
The objective of the neural net is then to predict how much time until the next stop.
My initial approach was to use a simple LSTM model that takes the event codes as input and produces a prediction of the time to the next sequence end. To handle time prediction, I normalized the time to stop using a log function in order to:
- Reduce the numbers to predict (since I calculate the time to stop in seconds)
- give higher penalties the closer we are to zero, e.g. [0 day - 1 day] is to be penalized more than [10 days - 11 days].
For training, I used a Mean Squared Error (MSE) loss function, AdamW optimizer with a learning rate of 1e-4, weight decay of 1e-2, and gradient clipping with a norm threshold of 0.5.
No matter the number of layers, hidden size, activations and dropout I tried, every network seems to just settle on always predicting the average time left of all sequences, with occasional spikes in either the up or down direction when some events happen.
I'm asking for suggestions on improving my model architecture and data normalization techniques. Any insights or recommendations, papers or other questions on the exchange would be greatly appreciated. Thank you all!
P.S. I've refrained from posting code since this is primarily a network design question. However, if sharing code snippets would help, please let me know. Also, I'm currently working with PyTorch 2.
