1
$\begingroup$

What is the concept of pruning a tree in Machine Learning regression problems? I am confused and a simple explanation would be great.

$\endgroup$

1 Answer 1

1
$\begingroup$

The concept of pruning a tree is mainly to avoid overfitting in either decision tree (DT) classifier or regressor learning when there is noise in the data or the number of training samples is too small to produce a representative sample of the true target function.

Furthermore DT regressor usually needs to transform possibly many features' continuous ranges to dynamically created possibly multiple ad hoc Boolean test attributes based on their information gains without firm theoretical basis, and each of its leaf nodes contains a constant value that serves as the predicted value which is typically the mean or median of the target values of the training data in each leaf node partition, therefore this effectively creates a noisy training dataset even the untransformed raw dataset is noise free. Thus for regression problem DT learning is very prone to overfitting and the concept of pruning becomes extremely important to reduce its inherent high variance and MSE/R-squared of its performance metrics.

Of course the learning algo can stop growing the tree earlier before it reaches the point where it perfectly regresses all the training data, but it's difficult to estimate precisely the minimum tree depth or leaf nodes.

Prepruning methods share a common problem, the horizon effect. This is to be understood as the undesired premature termination of the induction by the stop () criterion.

Thus post-pruning (or just pruning) is the most common way of simplifying trees. For DT regressors you just replace below classification accuracy confusions of validation data with MSE or R-squared

Here, nodes and subtrees are replaced with leaves to reduce complexity. Pruning can not only significantly reduce the size but also improve the classification accuracy of unseen objects... The procedures are differentiated on the basis of their approach in the tree (top-down or bottom-up)... One of the simplest forms of pruning is reduced error pruning. Starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept. While somewhat naive, reduced error pruning has the advantage of simplicity and speed.

For a detailed pioneering example of the popular reduced error pruning see Quinlan, J. R. (1987), 'Simplifying decision trees', International Journal of Man-Machine Studies 27, 221-234.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .