For Random Forests in particular, you will find a nice collection on references regarding research on the choice of number of trees in BiauScornet2015, section 2.4.
Further, a Random Forest can be considered an ensemble of decision trees. There is a bunch of literature on what, in fact, makes an ensemble effective. As far as I could gather, the gist of it seems to be that we need the right kind of "diversity" in the ensemble. Diversity is the keyword that seems to be used in the literature.
Coming from the practical side, you may find some research that investigates the relationship between ad-hoc/intuitive diversity measures (see also Zhou2012 sec 5.3) and ensemble performance for the classification task. The results: "it's complicated". [1, 2, 3]
However, from the theoretical side, I feel like there is plenty of motivation for this notion. A very good read is "Zhou2012: Ensemble Methods: Foundations and Algorithms". Let me quote two points from there in a condensed manner.
Error-Ambiguity Decomposition
Assume that the task is to use an ensemble of $T$ individual learners $h_1, \ldots, h_T$ to approximate a function $f: R^d \mapsto R$, and the final prediction of the ensemble is obtained through weighted averaging (4.9), i.e.,
$$
H(\boldsymbol{x})=\sum_{i=1}^T w_i h_i(\boldsymbol{x})
$$
where $w_i$ is the weight for the learner $h_i$, and the weights are constrained by $w_i \geq 0$ and $\sum_{i=1}^T w_i=1$
Let the errors of an individual learner (submodel) and the error of the ensemble be, resp.
$$
\begin{aligned}
& \operatorname{err}\left(h_i \mid \boldsymbol{x}\right)=\left(f(\boldsymbol{x})-h_i(\boldsymbol{x})\right)^2 \\
& \operatorname{err}(H \mid \boldsymbol{x})=(f(\boldsymbol{x})-H(\boldsymbol{x}))^2
\end{aligned}
$$
Likewise, we can define the ambiguity, a measure of disagreement among individual learners on instance (point / sample) $x$:
$$
\operatorname{ambi}\left(h_i \mid \boldsymbol{x}\right)=\left(h_i(\boldsymbol{x})-H(\boldsymbol{x})\right)^2,
$$
The generalization error and the ambiguity of the individual learner $h_i$ can be written respectively as
$$
\begin{aligned}
\operatorname{err}\left(h_i\right) & =\int \operatorname{err}\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x}, \\
\operatorname{ambi}\left(h_i\right) & =\int a m b i\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x} .
\end{aligned}
$$
The generalization error of the ensemble can be written as
$$
\operatorname{err}(H)=\int \operatorname{err}(H \mid \boldsymbol{x}) p(\boldsymbol{x}) d \boldsymbol{x}
$$
Based on the above notations, we can get the error-ambiguity decomposition [Krogh and Vedelsby, 1995]
$$
\operatorname{err}(H)=\overline{\operatorname{err}}(h)-\overline{a m b i}(h),
$$
where $\overline{\operatorname{err}}(h)=\sum_{i=1}^T w_i \cdot \operatorname{err}\left(h_i\right)$ is the weighted average of individual generalization errors, and $\overline{a m b i}(h)=\sum_{i=1}^T w_i \cdot \operatorname{ambi}\left(h_i\right)$ is the weighted average of ambiguities that is also referred to as the ensemble ambiguity.
This shows that the the more accurate and the more diverse the individual learners, the better the ensemble.
Bias-Variance-Covariance Decomposition
The bias-variance-covariance decomposition of squared error of ensemble is
$$
\operatorname{err}(H)=\overline{\operatorname{bias}}(H)^2+\frac{1}{T} \overline{\operatorname{variance}}(H)+\left(1-\frac{1}{T}\right) \overline{\operatorname{covariance}}(H) .
$$
where
\begin{aligned}
& \overline{\operatorname{bias}}(H)=\frac{1}{T} \sum_{i=1}^T\left(\mathbb{E}\left[h_i\right]-f\right) \\
& \overline{\operatorname{variance}}(H)=\frac{1}{T} \sum_{i=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right)^2, \\
& \overline{\operatorname{covariance}}(H)=\frac{1}{T(T-1)} \sum_{i=1}^T \sum_{j=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right) \mathbb{E}\left(h_j-\mathbb{E}\left[h_j\right]\right)
\end{aligned}
The smaller the covariance, the better the ensemble. It is obvious that if all the learners make similar errors, the covariance will be large, and therefore it is preferred that the individual learners make different errors. Thus, through the covariance term, (5.18) shows that the diversity is important for ensemble performance. Notice that the bias and variance terms are constrained to be positive, while the covariance termcan be negative
See Also
