0
$\begingroup$

Let's say that we have a test data set with $20,000$ observations for which we want to make a binary prediction for. When we apply our best trained model to this data set (e.g. logistic regression with threshold = 0.5, data_size = 4000 rows, 5 fold cv), only about $1 \%$ of the predictions are positive. That is, $p(\text{positive}) \geq 0.5$ is true only for about $1 \%$ of the predictions. We expect many more positives since the recall of the positive class of the best trained model is about $40 \%$. If we manually lower the threshold to $0.45$, then about $10 \%$ of the predictions are positive. Assume that the $20,000$ observations come from the same distribution as the training/validation data and are independent samples.

Questions.

  1. Why would a a model with decent recall for the positive class predict very few positives in the out of sample data?
  2. If (1) is true, then is it appropriate to lower the threshold for positive (e.g. $0.5$ to $0.45$) to increase the number of predicted positives in the test set?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.