0
$\begingroup$

I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not amazing. I have daily data covering 3 years, that I split to train/test with an 80/20% split. To try and improve results, I repeated the split however I did it randomly, i.e. on a shuffled data. My model, does not use any relations between data points, it only predicts the lead time using that specific sample's features (e.g. type of food, who's the carrier, expected delivery duration, etc.) When I do this, the results improve dramatically, which makes me question I'm doing something illegal. I wanted to know if I can actually shuffle the data? Can I do that?

$\endgroup$

2 Answers 2

1
$\begingroup$

Yes, shuffling the dataset is in fact, an important step in the process. Shuffling help in reducing overfitting to a particular pattern and generates more 'randomness' to the data, hence helping the model get used to variations and be more generalised.

Hence, you will want to shuffle to make sure that your training/test/validation sets are representative of the overall distribution of the data.

This link will give you more in-depth explanantions if you require them.

$\endgroup$
1
  • $\begingroup$ Thank you for the answer! As I wrote above to @Edoardo, when I test the model on a new chunk of data (even with shuffling) the performance is bad. I suspect it's because I have data for multiple vendors, types of food, etc. Is there a way of dealing with this issue? Some data drift detection, or continuous re-training? $\endgroup$
    – John Coles
    Jul 23, 2022 at 7:30
1
$\begingroup$

As a rule of thumb, you not just should but you must shuffle your training instances always. You just need to pay attention to not be fooled in regard to what a training instance is in your specific use case.

In the most classic situation you'll exactly what you did (which is right) and that's baffling you: given n training instances with n features you just randomly change the order of each training instance. So this dummy example

Feature 1 Feature 2
Instance 1 F1,1 F1,2
Instance 2 F2,1 F2,2
Instance 3 F3,1 F3,2

might turn into this

Feature 1 Feature 2
Instance 3 F3,1 F3,2
Instance 2 F2,1 F2,2
Instance 1 F1,1 F1,2

There's only one situation in which you need to pay attention, which is when you're dealing with time series. A time series is a sequence of repeated measurements, and we use them to train predictive models when we care about learning the temporal dependencies between events and their features.

In practice, if you measure the same event $n$ times gathering $m$ features you'll have $n*m$ element that compose the time serie. Note that the different time stamps in this case do not represent separate training instances, rather different time series are separate training instances. For example, I can record several weather features at different hours for a specific city, and that would give me a single time serie, by repeating the process for several cities I would gather different training instances. Once you understand the difference is pretty easy to see that you can still randomize your training instances, the only thing you can't randomize, for obvious reasons, is the order of the time stamps.

Time step 1 Time step 2
Feat 1 Feat 2 Feat 1 Feat 2
Instance 1 F1,1 F1,2 F1,1 F1,2
Instance 2 F2,1 F2,2 F2,1 F2,2
Instance 3 F3,1 F3,2 F3,1 F3,2

$\endgroup$
1
  • $\begingroup$ Thanks for the elaborate answer! So from the description of the problem, I understand I need to shuffle the data, however, to test the hypothesis, I left the last 3 months of data as test data. The rest was shuffled and used as training. I got bad performance on test. Some of the things I see is that vendors change, or type of food ordered. Perhaps I need to re-train every N test samples (using the last M)? Is there a good way of detecting a drift/change in data distribution? (I can try and build a model per vendor e.g. but that seems too much to handle...) $\endgroup$
    – John Coles
    Jul 23, 2022 at 7:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .