I have trained a XGboost model to predict survival for the Kaggle Titanic ML competition.
As with all Kaggle competitions there is a
train dataset with the target variable included and a
test dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.
I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the
train data. The accuracy I get from this validation is ~80% which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).
The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.
However when I submit my predictions for the
test set my public score is ~35% which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.
Why is my model so much worse on the
I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.
However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.
So can you help me understand how those two issues happen at the same time:
- Drastic difference between local accuracy and public score
- Reversing actually leads to the better public score.
Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):