Shuffling vs Non-shuffling train/test set yields drastically different results

Question

I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input to the NN I just pass the current state (1 time step), however that state is derived with some sort of scaling from past states.

Now I made a couple strange observation during training, I can't explain at all:

1. When I shuffle the whole data set and then do train/test split, the classification performance is extremely good, ~0.93 accuracy (3 classes) and identically generalizes onto the test set during the whole training
2. When I split the data into train/test and just shuffle train, the performance is less on train, but still acceptable (~0.75 accuracy), but performance on test falls off to ~0.36 accuracy
3. When I just split the data without shuffling, the train performance further drops to ~0.5 and test performance to ~0.3 (which is worse than randomly guessing)

What is going on here? I checked if I might have introduced some data leakage or lookahead bias in my data set manipulations, but couldn't find anything.

Anybody has an idea what is going on here?

Cheers

Answer 1

I don't have much information about your model and your data. But my best guess is that your dataset doesn't have the same distribution everywhere, and that without shuffling it before doing the split, you create subsets of data that are statistically very different i.e. their intrinsic distribution might differ a lot. When you shuffle before splitting, you mitigate this effect because you distribute your data (which might follow divergent patterns according to timestamps or any other feature) in both sets.