Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I have been following the ML course by Tom Mitchel. The inherent assumption while using Decision Tree Learning Algo is: The algo. preferably chooses a Decision Tree which is the smallest.
Why is this so when we can have bigger extensions of the tree which could in principle perform better than the shorter tree?
Consider the LCD (least common denominator) principle in algebra. A larger denominator would work for most processes for which the LCD would be calculated, however the least is the one used by convention. Why? The interest in prime numbers in general is based on the value of reductive methodologies.
In philosophy, Occam's Razor is a principle that, given two conceptual constructs that correlate equally well with observations, the simpler is most likely the best. A more formal and generalized prescription is this:
Given two mathematical models of a physical system with equal correlation to the set of all observations about the system being modeled, the simpler model is more likely to be the most likely to be predictive of conditions outside those of the observation set.
This principle of simplicity as the functional ideal is true of decision trees. The more likely functional and clear decision tree resulting from a given data set driving its construction will be the simplest. Without a clear reason for additional complexity, there may be no benefit derived from the complexity added and yet there may be penalties in terms of clarity and therefore maintainability and verifiability. There may be computational penalties too, in some applications.
The question mentions, "Bigger extensions of the tree which could in principle perform better," however the performance of the tree should be a matter optimization in preparing for execution and execution of the decision tree in real time. In other words, the minimalist decision tree is the most clear, workable, and verifiable construct, however a clever software engineer could translate that minimalist construct to a run time optimized equivalent.
Just as with compilation of source code, performance is multidimensional in meaning. There are time efficiency, memory efficiency, network bandwidth efficiency, or other performance metrics that could be used when optimizing the tree for run time. Nonetheless, the simpler tree is the best starting point for any weighted combination of these interests.
The bigger your tree is the more overfitting your model is. In machine learning, we always prefer a simpler model unless there is good reason to go for complication.
Adding to SmallChess's answer , Larger trees(with many nodes) are too adapted to the training set, as a small change in the input train data might cause the trees to change very much and hence change the estimate value too much.This is mainly due to the hierarchical structures of trees(because a change in a higher node may cause all lower nodes to change). As an extreme case you can think of a large tree where in each training example has its own node.Such a is absolutely useless for test prediction.
