Suppose we have a classical machine learning problem. Say $m$ training examples and $n$ features with $m >> n$. Suppose I find a great algorithm using automl or otherwise that gives 95% accuracy on the training set. What prevents us from learning from the 5% mistakes (if $0.05 m >>1$) and training a new algorithm just on these 5% mistakes using another algorithm and so on till we get a small sample when this can't be done anymore?

  • 3
    $\begingroup$ To begin with, we don't really care about training accuracy, but rather, we want the model to generalize and perform well on unseen samples. Maximizing training accuracy is trivial anyway: just build a hashmap from samples to labels. $\endgroup$
    – SpiderRico
    Oct 28, 2023 at 21:54
  • $\begingroup$ Isn’t this the basis of boosting? $\endgroup$
    – Dave
    Oct 31, 2023 at 2:49

1 Answer 1


Getting a high training accuracy on data without contradictions is nearly always possible, given a parametric function that can fit it in theory (enough flexibility and free parameters). It often does not require any special handling, such as the divide and conquer approach suggested in the question. Although that could possibly work.

However, copying the training data accurately as possible is not the ultimate goal for most ML. Instead the more normal goal is to use the trained model to predict and make use of output values when they are not otherwise known - e.g. they predict future events, or things that take more effort to measure/calculate than the input data. For these kinds of use, the performance of ML on the training data is secondary to its performance on data outside of the training set.

Forcing a perfect match to training data often leads to a phenomenon called overfitting, where performance on new unseen data is far worse than performance on the training data. So usually ML researchers focus their effort on improving results against validation and ultimately test data, which is data in the same format as training data including known target values. As the point is to not train against this data, it is not possible to force accuracy in the same way, and doing so would be counterproductive anyway: you want to know how good the ML is when used in reality, not just get a high score in a virtual space


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .