Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?

Question

Based on the answer of my previous question: How can I avoid overfitting when doing parameter tuning?

Can we say: the more we increase the numbers K of cross validation the less likely it is that we overfit?

Answer 1

In general, no.

There is a tradeoff between making the validation set for each fold smaller, and having more folds in total.

As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance of the model. In fact, you can construct examples where the error is arbitrarily large.

For this reason, people sometimes use Bootstrap Validation if they need a very large number of folds. In practice though, most people just us 10 folds, and that's "good enough".