2
$\begingroup$

In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result are removed, and I think that this technique is also called "dropout". Am I correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this.

In your decision tree example, I believe you're talking about pruning, which is different. In that context you're removing nodes that you know aren't contributing. In dropout, you randomly turn off nodes during training in order to prevent individual nodes from being too influential, but the nodes are never removed.

You might also be interested in L1 regularization in parameterized models. This is when you add a penalty according to the absolute weights (rather than square weights), which tends to drive less useful weights to 0. Then you can remove the nodes with almost no weight. This is more akin to your decision tree example rather than dropout though.

Improve this answer
$\endgroup$
4
  • $\begingroup$ It may be also worth mentioning "pruning" in neural networks, as the OP may come across it too. $\endgroup$
    – nbro
    Commented Jul 18, 2021 at 13:12
  • $\begingroup$ Ah good point, I forgot about that. Admittedly I don't know much about neural net pruning, I would've guessed it's similar to l1. $\endgroup$
    – harwiltz
    Commented Jul 18, 2021 at 13:18
  • 1
    $\begingroup$ xgboost (a decision-tree based algorithm) does use random feature selection and random row selection for different trees, which is more similar to dropout than pruning. $\endgroup$
    – Neil Slater
    Commented Jul 18, 2021 at 16:34
  • 1
    $\begingroup$ When Random Forest is constructed features of each individual tree are selected at Random, (or in other words dropped out at random), this is done so that the overall model is not too dependent on one feature. This is pretty similar to dropout in Neural Networks. $\endgroup$
    – Akavall
    Commented Jul 18, 2021 at 20:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .