9 votes
Accepted

Are decision tree learning algorithms deterministic?

Are decision tree learning algorithms deterministic? Given a fixed dataset, do they always produce a tree with the same structure? Generally, yes. Most decision tree learners, like the common ID3 and ...
John Doucette's user avatar
8 votes
Accepted

Why does nobody use decision trees for visual question answering?

For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better ...
Greenstick's user avatar
5 votes
Accepted

Why isn't my decision tree classifier able to solve the XOR problem properly?

I can reproduce this problem for an even more easily separable dataset: The ideal tree for it should be as follows: However, when I run DecisionTreeClassifier ...
Vladislav Gladkikh's user avatar
4 votes
Accepted

What do the values of the leaves of the decision tree represent?

Decision tree nodes are split bases on the number of data samples, these numbers indicate the number of data samples they are fit to. In your case samples = 256. It is further split into two nodes of ...
thecomplexitytheorist's user avatar
4 votes

How could decision tree learning algorithms cope with imbalanced classes?

Decision Tree learners, on their own, are not a good way to deal with imbalanced data. The most commonly used algorithms, by default, make no attempt to address this problem. If you look carefully at ...
John Doucette's user avatar
4 votes
Accepted

Why are decision trees and random forests scale invariant?

Scaling only makes sense when there is something that reacts to that scale. Decision Trees though, just make a cut at a certain number. Imagine: For a feature that goes from 0 to 100 a cut at 50 may ...
N. Kiefer's user avatar
  • 321
4 votes
Accepted

Is the dropout technique specific only to neural networks?

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal ...
harwiltz's user avatar
  • 1,126
3 votes
Accepted

Why is the exponential loss used in this case?

The loss is $$\mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right),$$ which can also be written ...
nbro's user avatar
  • 40.1k
3 votes
Accepted

How does the decision tree implicitly do feature selection?

Consider a dataset $S \in \mathbb{R}^{N \times (M + 1)}$ with $N$ observations (or examples), where each observation $S_i \in \mathbb{R}^{M + 1}$ is composed of $M$ elements, one value for each of the ...
nbro's user avatar
  • 40.1k
3 votes
Accepted

What makes a machine learning algorithm a low variance one or a high variance one?

What this is talking about is how much a machine learning algorithm is good at "memorizing" the data. Decision trees, for their nature, tend to overfit very easily, this is because they can separate ...
user's user avatar
  • 146
3 votes
Accepted

Why was Go a harder game for an AI to master than Chess?

The branching factor is important, as it limits the effectiveness of search. However, the branching factor in chess is already too high to effectively search without techniques that reduce the size ...
Neil Slater's user avatar
  • 31.5k
3 votes

Decision tree: more than 2 classes, how to represent elements that are in a class vs ones that aren't?

I don't think that is possible with a decision tree, unless there is some measure of confidence that you can use as a threshold. I ran into the same problem with the ID3 algorithm. It assigns classes,...
Oliver Mason's user avatar
  • 5,397
3 votes

How to calculate the entropy in the ID3 decision tree algorithm?

Suppose you have data: ...
dynrepsys's user avatar
  • 1,363
3 votes
Accepted

Given a dataset with no noisy examples, is the training error for the ID3 algorithm always 0?

Yes, if you can assume that your data is separable on the features given then ID3 will find a decision tree for it (Note: this will not necessarily be an optimal tree, or even a good tree). To ...
Andrew Butler's user avatar
2 votes
Accepted

What is the intuition behind the entropy formula used in the ID3 algorithm?

Please, take a look at Understanding Shannon's Entropy metric for Information. The answer for the minus sign is in section 6. The probability logs are less than or equal to $0$, so the minus sign ...
xboard's user avatar
  • 176
2 votes

Why are tree-based models more widely used in Medical Diagnosis?

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article. It presents a case study of a hospital whose policy was to send ...
QuadraticOne's user avatar
2 votes

“Outside-in” versus “Inside-out” machine learning

It sounds like you are interested in the ideas of intrinsic motivation and attention in the context of machine learning. These are big topics, and the subject of much active research. Intrinsic ...
John Doucette's user avatar
2 votes
Accepted

What are possible functions assigned on decision nodes for decision tree prediction?

For a binary split, there are only three possible operations (or arguably only two if you consider one-hot encoding). Any other kind of split would simply not be binary. Almost every tree-based model ...
Frans Rodenburg's user avatar
2 votes

At which point we have to stop post pruning in decision tree?

There are a variety of conditions we can use when deciding whether to prune a sub-tree or not after generating a decision tree model. There are three common approaches. We can prune branches with ...
John Doucette's user avatar
2 votes

Why are decision trees and random forests scale invariant?

Feature scaling happens to be a problem when a model is characterized by having a distance metric (or another kind of numerical evaluation for that matter). Therefore models such as support vector ...
Ghostpunk's user avatar
2 votes

Why isn't my decision tree classifier able to solve the XOR problem properly?

The algorithm fails because it is greedy. This means that it takes the first split decision immediately, without taking into account what will happen in next steps. An alternative would be given by ...
Jaume Oliver Lafont's user avatar
2 votes

What is the most suitable AI technique to use for path planning?

If I had to implement a path exploration/finding algorithm on a robot, I would follow these steps: Make sure you can detect your position. You need to be able to record your position otherwise you ...
Alvin Sartor's user avatar
2 votes
Accepted

What is the difference between Inductive Learning and Connectionist Learning?

All of the statistical learning is about inductive learning. What is the difference between inductive learning and connectionist learning? Inductive learning is about identifying patterns from ...
Maheshwar Ligade's user avatar
2 votes
Accepted

Feature extraction timeseries, model compatibility

there's a lot to un-pack in this question. Why do they only pick 500 rows? my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features. note that ...
user1269942's user avatar
2 votes

Does the root node of a Decision Tree change when one data point is removed?

I would agree with you that (b) is the right answer. The root attribute splits the data set into two segments, usually based on entropy or some other metric. Removing a data point will probably change ...
Oliver Mason's user avatar
  • 5,397
1 vote
Accepted

Mathematical calculation behind decision tree classifier with continuous variables

Of course, it depends on what algorithm you use. Typically, a top-down algorithm is used. You gather all the training data at the root. The base decision is going to be whatever class you have most of....
Robby Goetschalckx's user avatar
1 vote

What are some applications where tree models perform better than neural networks?

Hard to say in general. Speaking from my own experience and by looking at which models win Kaggle competitions (see here and here), I would say tree-based models e.g. Random Forests, Decision Trees, ...
Tinu's user avatar
  • 628
1 vote

Can I apply AdaBoost on a random forest?

The random forest (rf) is the perfectly parallel ensemble of CART learners. It uses Gini impurity to inform its split locations, and ensemble summary (mean, mode) to track error. The gradient boosted ...
EngrStudent's user avatar
1 vote

What is the meaning of test data set in naive bayes classifier or decision trees?

In machine learning, we can use all the datasets as training data in a model. But if there are too many data sets, or too much data, and we do not split them up, our model may be not produce ...
Muhammad Mukhtar's user avatar
1 vote

What is the meaning of test data set in naive bayes classifier or decision trees?

Your assumption about the test data is not correct completely. Maybe you use the test data to tune your learning algorithm to work better on the test data, but it's not the whole thing. Sometimes you ...
OmG's user avatar
  • 1,816

Only top scored, non community-wiki answers of a minimum length are eligible