I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the
scikit-optimize package. For those unfamiliar with the package, it can be used easily with model classes from
scikit-learn, in this case the random forest classes such as
RandomForestClassifier, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.
I noticed that in some examples, the
n_estimators hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect. The
n_estimators hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in
scikit-optimize, only single variables).
Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model? If so, what is the rationale?