1
$\begingroup$

In the original paper on dropout, in section 7.3.2, we see that while keeping $pn$ constant, we get a (test) error increase by decreasing retainment below 0.6. Why would that happen? If $pn$ is constant, the capacity of the network should be constant, correct?

$\endgroup$

1 Answer 1

1
$\begingroup$

In the images below (figure 9 from section 7.3) $n$ refers to a unit in the network and $p$ refers to the probability of retaining a unit in the network. The combination of both, $pn$ thus refers to the "effective" number of units. The left image thus shows how training and test error change when varying the probability of retaining a unit ($p$) whereby the number of units ($n$) is fixed. The right image shows how the errors change when the "effective" number of units ($pn$) is fixed, and the amount of units ($n$) is adopted with an increasing $p$.

The test error is U-shaped with a sweet spot around 0.5 or 0.6 in both cases. In the left image where the number of units ($n$) is fixed, this likely occurs since the ensemble-like properties of dropout networks do not emerge with too little dropout ($p$ below 0.3) or too much dropout ($p$ above 0.8). Those results could however be purely due to a change in capacity. The right image keeps the "capacity" constant and only looks at the "pure dropout effect" since the capacity (namely the number of units $pn$) is fixed. This rules out a decrease in capacity as the driving factor. The interaction of the decrease in capacity due to an increase in dropout is hinted at with the following sentence:

we notice that the magnitude of errors for small values of p has reduced by a lot compared to Figure 9a

enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .