1
$\begingroup$

I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the package, it can be used easily with model classes from scikit-learn, in this case the random forest classes such as RandomForestClassifier, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.

I noticed that in some examples, the n_estimators hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect. The n_estimators hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in scikit-optimize, only single variables).

Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model? If so, what is the rationale?

$\endgroup$
1
  • 1
    $\begingroup$ Any time there is randomness, there is a chance the estimator could produce better results. $\endgroup$ Commented Jan 23, 2022 at 3:31

2 Answers 2

1
$\begingroup$

I would say that in general situation more estimators are better.

RandomForest fits a lot of estimators - decision trees that take a subset of data (obtained sampling with replacement) and subset of features (by default sqrt(n_features) in sklearn).

Each of these estimators is noisy and prone to overfitting, producing a complicated decision surface.

But when you take sufficiently many of them, noisy artifacts, produced by individual estimators, are smoothed and you can get pretty accurate classifier or regressor.

It can be the case, that some added estimators are too noisy and worsen the ensemble, but overall, quality is expected to improve. At some point, the amount of estimators would be sufficient, and additional change won't change the result a lot.

$\endgroup$
0
$\begingroup$

For Random Forests in particular, you will find a nice collection on references regarding research on the choice of number of trees in BiauScornet2015, section 2.4.

Further, a Random Forest can be considered an ensemble of decision trees. There is a bunch of literature on what, in fact, makes an ensemble effective. As far as I could gather, the gist of it seems to be that we need the right kind of "diversity" in the ensemble. Diversity is the keyword that seems to be used in the literature.

Coming from the practical side, you may find some research that investigates the relationship between ad-hoc/intuitive diversity measures (see also Zhou2012 sec 5.3) and ensemble performance for the classification task. The results: "it's complicated". [1, 2, 3]

However, from the theoretical side, I feel like there is plenty of motivation for this notion. A very good read is "Zhou2012: Ensemble Methods: Foundations and Algorithms". Let me quote two points from there in a condensed manner.

Error-Ambiguity Decomposition

Assume that the task is to use an ensemble of $T$ individual learners $h_1, \ldots, h_T$ to approximate a function $f: R^d \mapsto R$, and the final prediction of the ensemble is obtained through weighted averaging (4.9), i.e., $$ H(\boldsymbol{x})=\sum_{i=1}^T w_i h_i(\boldsymbol{x}) $$ where $w_i$ is the weight for the learner $h_i$, and the weights are constrained by $w_i \geq 0$ and $\sum_{i=1}^T w_i=1$

Let the errors of an individual learner (submodel) and the error of the ensemble be, resp. $$ \begin{aligned} & \operatorname{err}\left(h_i \mid \boldsymbol{x}\right)=\left(f(\boldsymbol{x})-h_i(\boldsymbol{x})\right)^2 \\ & \operatorname{err}(H \mid \boldsymbol{x})=(f(\boldsymbol{x})-H(\boldsymbol{x}))^2 \end{aligned} $$

Likewise, we can define the ambiguity, a measure of disagreement among individual learners on instance (point / sample) $x$: $$ \operatorname{ambi}\left(h_i \mid \boldsymbol{x}\right)=\left(h_i(\boldsymbol{x})-H(\boldsymbol{x})\right)^2, $$

The generalization error and the ambiguity of the individual learner $h_i$ can be written respectively as $$ \begin{aligned} \operatorname{err}\left(h_i\right) & =\int \operatorname{err}\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x}, \\ \operatorname{ambi}\left(h_i\right) & =\int a m b i\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x} . \end{aligned} $$

The generalization error of the ensemble can be written as $$ \operatorname{err}(H)=\int \operatorname{err}(H \mid \boldsymbol{x}) p(\boldsymbol{x}) d \boldsymbol{x} $$

Based on the above notations, we can get the error-ambiguity decomposition [Krogh and Vedelsby, 1995] $$ \operatorname{err}(H)=\overline{\operatorname{err}}(h)-\overline{a m b i}(h), $$

where $\overline{\operatorname{err}}(h)=\sum_{i=1}^T w_i \cdot \operatorname{err}\left(h_i\right)$ is the weighted average of individual generalization errors, and $\overline{a m b i}(h)=\sum_{i=1}^T w_i \cdot \operatorname{ambi}\left(h_i\right)$ is the weighted average of ambiguities that is also referred to as the ensemble ambiguity.

This shows that the the more accurate and the more diverse the individual learners, the better the ensemble.

Bias-Variance-Covariance Decomposition

The bias-variance-covariance decomposition of squared error of ensemble is $$ \operatorname{err}(H)=\overline{\operatorname{bias}}(H)^2+\frac{1}{T} \overline{\operatorname{variance}}(H)+\left(1-\frac{1}{T}\right) \overline{\operatorname{covariance}}(H) . $$ where \begin{aligned} & \overline{\operatorname{bias}}(H)=\frac{1}{T} \sum_{i=1}^T\left(\mathbb{E}\left[h_i\right]-f\right) \\ & \overline{\operatorname{variance}}(H)=\frac{1}{T} \sum_{i=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right)^2, \\ & \overline{\operatorname{covariance}}(H)=\frac{1}{T(T-1)} \sum_{i=1}^T \sum_{j=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right) \mathbb{E}\left(h_j-\mathbb{E}\left[h_j\right]\right) \end{aligned}

The smaller the covariance, the better the ensemble. It is obvious that if all the learners make similar errors, the covariance will be large, and therefore it is preferred that the individual learners make different errors. Thus, through the covariance term, (5.18) shows that the diversity is important for ensemble performance. Notice that the bias and variance terms are constrained to be positive, while the covariance termcan be negative

See Also

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .