I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input to the NN I just pass the current state (1 time step), however that state is derived with some sort of scaling from past states.

Now I made a couple strange observation during training, I can't explain at all:

  1. When I shuffle the whole data set and then do train/test split, the classification performance is extremely good, ~0.93 accuracy (3 classes) and identically generalizes onto the test set during the whole training
  2. When I split the data into train/test and just shuffle train, the performance is less on train, but still acceptable (~0.75 accuracy), but performance on test falls off to ~0.36 accuracy
  3. When I just split the data without shuffling, the train performance further drops to ~0.5 and test performance to ~0.3 (which is worse than randomly guessing)

What is going on here? I checked if I might have introduced some data leakage or lookahead bias in my data set manipulations, but couldn't find anything.

Anybody has an idea what is going on here?


  • $\begingroup$ How are you shuffling your data? For time series not every shuffling is conceptually right, see this answer for an explanation $\endgroup$ Aug 14, 2022 at 23:14

1 Answer 1


I don't have much information about your model and your data. But my best guess is that your dataset doesn't have the same distribution everywhere, and that without shuffling it before doing the split, you create subsets of data that are statistically very different i.e. their intrinsic distribution might differ a lot. When you shuffle before splitting, you mitigate this effect because you distribute your data (which might follow divergent patterns according to timestamps or any other feature) in both sets.

  • $\begingroup$ That could definitely make sense. I had a similar thought and compared the distribution of the features (in fact a histogram since they are categorical) between train and test. The plots look slightly different but not tremendously. However, I have to admit that they could be more insightful. Do you have an idea how to best approach such a statistical comparison and what metrics to look out for? A very simple approach could just be DataFrame.describe(). $\endgroup$
    – NicFit_88
    Aug 15, 2022 at 8:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .