Does a bias also have a chance to be dropped out in Dropout layer?

Question

Suppose that you have 80 neurons in a layer, where one neuron is bias. Then you add a dropout layer after the activation function of this layer.

In this case, does it have a chance to drop out the bias neuron, or does the dropout only affect the other 79 weight neurons?

Answer 1

Only the non-bias ones,

It is discouraged to include the bias weights under norm penalty regularization for example, so why should it be included in the drop-out regularization scheme?

Drop out can be implemented by multiplying units with zero and the bias term is rather special. The bias term determines the distance from the origin the linear decision boundary of the node implements. It is included in every sum, but it does not receive any inputs.

So to take your example, say you have 80 nodes per layer in an MLP, where one is a bias node, the output of these layers will consist of 79 nodes every time. This output is then put under the drop-out effect.

From Goodfellows et al. Deep Learning book under the regularization chapter, present online: "http://www.deeplearningbook.org/contents/regularization.html", they write that they implement the drop-out by canceling some outputs to zero:

"
In most modern neural networks, ... , we can effectively remove a unit from a network by multiplying its output value by zero. ... . Here, we present the dropout algorithm in terms of multiplication by zero for simplicity, but it can be trivially modified to work with other operations that remove a unit from the network.
"

Drop-out by cancelling outputs, the output from the activation function, has nothing to do with the bias vector. It is not receiving any inputs. Therefore I think the implementation typically only deals with output-to-input connections, and the bias vector does not receive inputs so it can safely be let alone from the drop-out process.