1
$\begingroup$

I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the package, it can be used easily with model classes from scikit-learn, in this case the random forest classes such as RandomForestClassifier, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.

I noticed that in some examples, the n_estimators hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect. The n_estimators hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in scikit-optimize, only single variables).

Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model? If so, what is the rationale?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Any time there is randomness, there is a chance the estimator could produce better results. $\endgroup$
    – FourierFlux
    Jan 23 at 3:31

1 Answer 1

Reset to default
1
$\begingroup$

I would say that in general situation more estimators are better.

RandomForest fits a lot of estimators - decision trees that take a subset of data (obtained sampling with replacement) and subset of features (by default sqrt(n_features) in sklearn).

Each of these estimators is noisy and prone to overfitting, producing a complicated decision surface.

But when you take sufficiently many of them, noisy artifacts, produced by individual estimators, are smoothed and you can get pretty accurate classifier or regressor.

It can be the case, that some added estimators are too noisy and worsen the ensemble, but overall, quality is expected to improve. At some point, the amount of estimators would be sufficient, and additional change won't change the result a lot.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .