I know the random forest is a bagging technique. But what if my random forest overfits on a dataset, so I reduce the depth of the decision tree and now it is underfitting. In this scenario, can I take the under-fitted random forest with little depth and try to boost it?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Random forest does not overfit. Check this - stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#remarks $\endgroup$– naiveCommented Jul 17, 2020 at 13:06
-
$\begingroup$ @naive - any learner can overfit, though the RF tries hard not to do so. $\endgroup$– EngrStudentCommented Dec 28, 2022 at 21:09
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The random forest (rf) is the perfectly parallel ensemble of CART learners. It uses Gini impurity to inform its split locations, and ensemble summary (mean, mode) to track error.
The gradient boosted machine (gbm) is the perfectly serial ensemble of CART learners. It can use the Adaboost method of weighting error, though gradient boosting tends to be preferred. The first tree is the predictor, and all others try and resolve the error.
There is a multiplicity of ensembles that are not 100% parallel or 0% parallel.
I have had decent results by predicting using the RF, computing the Error, and then using a gbm to try and clean up the error. It is something that has been "on the menu" for ~20 years since those learners came out.
EDIT:
These folks talk about it some. I don't know if they made a random-forest of gradient boosted trees, or a gradient boosted tree of random-forests (its been a while since I went through it).
