# Why is my model accuracy high in train-test split but actually worse than chance in validation set?

I have trained a XGboost model to predict survival for the Kaggle Titanic ML competition.

As with all Kaggle competitions there is a train dataset with the target variable included and a test dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.

My problem:

I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the train data. The accuracy I get from this validation is ~80% which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).

The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.

However when I submit my predictions for the test set my public score is ~35% which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.

Why is my model so much worse on the test?

I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.

However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.

So can you help me understand how those two issues happen at the same time:

• Drastic difference between local accuracy and public score
• Reversing actually leads to the better public score.

Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):

https://www.kaggle.com/fnguyen/titanicrising-test

• That link leads to a 404 error. – Brian Spiering Feb 3 at 2:17
• @BrianSpiering Yes sorry, the notebook wasn't public but it should be now. – Fnguyen Feb 3 at 13:47

Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance.

It is best practices to put all data transformations in a function so they can be applied to all data in a similar way.

Since you are using scikit-learn, sklearn.compose.ColumnTransformer is designed for this purpose. Example code for the Titanic dataset is here.

• I will try that but functionally the data transformation for both datasets is the same with the only difference being that the test data does not contain a "Ms" at factor level. Nevertheless you never know what makes a quirk in the data so I will package all transformations in a function and apply the same to both sets to eliminate this factor. – Fnguyen Feb 3 at 14:12
• I have cleaned up my code using your answer and a notebook I found on Kaggle as inspiration. By joining test and train dataset first, then doing all necessary transformations and only after splitting them again for modeling and prediction I was able to remove the error. They key to his was actually that I was applying an Imputation Method, I will describe that in an answer in detail. – Fnguyen Feb 5 at 14:19

@BrianSpiering was generally correct in pointing out that you should always apply the same transformations to your train and test dataset.

This was the key to my solution which was a bit more specific but might actually help others who encounter a similar problem.

Specifically my mistake came about because of Imputation! Some of the factors I used for my model were NA in both the train and test data set. To complete the data I simply imputed these missings using mean and mode respectively. However since I did those transformations separately on both sets the actual mean/mode value that was used differed heavily! By applying the imputation on the full data I also imputed the same data for all missing cases which solved my error.

My resulting accuracy in the public leaderboard is now at 74.2% which is fairly close to my local test score of 79.6%.