# Tag Info

8

Are decision tree learning algorithms deterministic? Given a fixed dataset, do they always produce a tree with the same structure? Generally, yes. Most decision tree learners, like the common ID3 and C4.5/C5.0 algorithms, are deterministic. At each step, the learners consider all possible features that have not yet been used to split the data and find the ...

6

For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better to inputs and maintain performance when inputs have undergone certain linear transformations (e.g. some scaling or a translation along the x-axis). These ...

4

Decision tree nodes are split bases on the number of data samples, these numbers indicate the number of data samples they are fit to. In your case samples = 256. It is further split into two nodes of 154 and 102.

4

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this. In your decision tree example, I believe you're talking about pruning, which is different. In that context you're ...

4

Decision Tree learners, on their own, are not a good way to deal with imbalanced data. The most commonly used algorithms, by default, make no attempt to address this problem. If you look carefully at the three sources you post, you will find that they actually all agree on this point. Two of the sources actually propose methods of addressing this ...

3

I can reproduce this problem for an even more easily separable dataset: The ideal tree for it should be as follows: However, when I run DecisionTreeClassifier with the maximal depth = 2 in scikit-learn many times, it splits the dataset randomly and never gets it right. This is an example of 4 different runs: The problem is that scikit-learn has only two ...

3

The branching factor is important, as it limits the effectiveness of search. However, the branching factor in chess is already too high to effectively search without techniques that reduce the size of the search space. Even with millions of tests per second, a computer can only check a small fraction of the possible future games in order to find results in ...

3

I don't think that is possible with a decision tree, unless there is some measure of confidence that you can use as a threshold. I ran into the same problem with the ID3 algorithm. It assigns classes, but you only have the resulting class without any confidence or probability attached. One possible solution could be to add a number of counter examples as a ...

3

Suppose you have data: color height quality ===== ====== ======= green tall good green short bad blue tall bad blue short medium red tall medium red short medium To calculate the entropy for quality in this example: X = {good, medium, bad} x1 = {good}, x2 = {bad}, x3 = {medium} Probability of each x in X: p1 = 1/6 = 0....

3

Yes, if you can assume that your data is separable on the features given then ID3 will find a decision tree for it (Note: this will not necessarily be an optimal tree, or even a good tree). To understand why let's look at a proof. Assume we have one feature left and some number of examples in a leaf that does not have separated data points then, either: ...

2

Please, take a look at Understanding Shannon's Entropy metric for Information. The answer for the minus sign is in section 6. The probability logs are less than or equal to $0$, so the minus sign guarantees that information (entropy) is always greater than or equal to $0$.

2

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article. It presents a case study of a hospital whose policy was to send asthma sufferers to an intensive care unit; the intensive care meant they were less likely to develop pneumonia and therefore the data showed that people with ...

2

If I had to implement a path exploration/finding algorithm on a robot, I would follow these steps: Make sure you can detect your position. You need to be able to record your position otherwise you have no reference for the exploration. You don't need a global positioning system (like GPS), a local one is more than enough in your case. This means that the ...

2

All of the statistical learning is about inductive learning. What is the difference between inductive learning and connectionist learning? Inductive learning is about identifying patterns from examples. It is more related to statistics. Connectionist learning is more about finding a common pattern and predicting as well as self-learning(learning from the ...

2

there's a lot to un-pack in this question. Why do they only pick 500 rows? my guess: in order to keep the example running quickly. tsfresh usually takes a while to calculate its features. note that when they evaluated their model, they took the last 500 samples. What's the point of re-arranging the rows/columns? answer: the data frame format that ...

2

Consider a dataset $S \in \mathbb{R}^{N \times (M + 1)}$ with $N$ observations (or examples), where each observation $S_i \in \mathbb{R}^{M + 1}$ is composed of $M$ elements, one value for each of the $M$ features (or independent variables), $f_1, \dots f_M$, and the corresponding target value $t_i$. A decision tree algorithm (DTA), such as the ID3 ...

2

For a binary split, there are only three possible operations (or arguably only two if you consider one-hot encoding). Any other kind of split would simply not be binary. Almost every tree-based model is restricted to binary splits, due to the combinatorial explosion when considering ternary or even more complex splits. Of course you could write your own ...

2

There are a variety of conditions we can use when deciding whether to prune a sub-tree or not after generating a decision tree model. There are three common approaches. We can prune branches with less support than a specific threshold. These are branches which were constructed using very few points from the training data. We can prune branches where the ...

2

What this is talking about is how much a machine learning algorithm is good at "memorizing" the data. Decision trees, for their nature, tend to overfit very easily, this is because they can separate the space along very non-linear curves, especially if you get a very deep tree. Simpler algorithms, on the other hand, tend to separate the space along linear ...

2

The loss is $$\mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right),$$ which can also be written as follows $$\mathcal{L} = \sum_{i=1}^{N} e^{-y_{i} f\left(\mathbf{x}_{i}\right)} \tag{1}\label{1}$$ The important thing to note here is the $-$ in the exponent, ...

2

Feature scaling happens to be a problem when a model is characterized by having a distance metric (or another kind of numerical evaluation for that matter). Therefore models such as support vector machines, neural networks, distance based clustering methods (e.g. k means) and linear/logistic regression are prone to changes by feature scaling. Those which are ...

2

Scaling only makes sense when there is something that reacts to that scale. Decision Trees though, just make a cut at a certain number. Imagine: For a feature that goes from 0 to 100 a cut at 50 may be improving performance. Scaling this down to 0 to 1 making the cut a 0.5 doesn't change a thing. Now on the other hand NN have some kind of activation function ...

1

Of course, it depends on what algorithm you use. Typically, a top-down algorithm is used. You gather all the training data at the root. The base decision is going to be whatever class you have most of. Now, we see if we can do better. We consider all possible splits. For categorical variables, every value gets its own node. For continuous variables, we can ...

1

Hard to say in general. Speaking from my own experience and by looking at which models win Kaggle competitions (see here and here), I would say tree-based models e.g. Random Forests, Decision Trees, Gradient Boosting are favorable over neural networks when working with low-dimensional data and easy interpretable features (usually simple tabular data with ...

1

In machine learning, we can use all the datasets as training data in a model. But if there are too many data sets, or too much data, and we do not split them up, our model may be not produce acceptable results. Why? Because if the model studies too much training data, it may be overfitted. (Just like when you cram for a test, and get overloaded with ...

1

Your assumption about the test data is not correct completely. Maybe you use the test data to tune your learning algorithm to work better on the test data, but it's not the whole thing. Sometimes you need to know that the ML method is working or not and have a sense about how much does it work! You have other scenarios that you want to evaluate your method: ...

1

It sounds like you are interested in the ideas of intrinsic motivation and attention in the context of machine learning. These are big topics, and the subject of much active research. Intrinsic motivation says that the key to identifying interesting patterns and skills that are worth learning is to give the agent some intrinsic reason to learn to do new ...

1

The number of estimators in Random Forest is a hyper-parameter. If you are using SKLearn's Random Classifier you can use one of the following techniques to find a (near) optimal hyperparameter settings (Note:You can tweak other hyperparameters like min_leaf_size etc as well with this approach); GridSearchCV You can specify a grid of all the hyperparameters ...

Only top voted, non community-wiki answers of a minimum length are eligible