I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize
package. For those unfamiliar with the package, it can be used easily with model classes from scikit-learn
, in this case the random forest classes such as RandomForestClassifier
, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.
I noticed that in some examples, the n_estimators
hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect. The n_estimators
hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in scikit-optimize
, only single variables).
Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model? If so, what is the rationale?