I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input to the NN I just pass the current state (1 time step), however that state is derived with some sort of scaling from past states.
Now I made a couple strange observation during training, I can't explain at all:
- When I shuffle the whole data set and then do train/test split, the classification performance is extremely good, ~0.93 accuracy (3 classes) and identically generalizes onto the test set during the whole training
- When I split the data into train/test and just shuffle train, the performance is less on train, but still acceptable (~0.75 accuracy), but performance on test falls off to ~0.36 accuracy
- When I just split the data without shuffling, the train performance further drops to ~0.5 and test performance to ~0.3 (which is worse than randomly guessing)
What is going on here? I checked if I might have introduced some data leakage or lookahead bias in my data set manipulations, but couldn't find anything.
Anybody has an idea what is going on here?