Does the input layer have bias and are there bias neurons?

Question

I have seen two different representations of neural networks when it comes to bias. Consider a "simple" neural network, with just an input layer, a hidden layer and an output layer. To compute the value of a neuron in the hidden layer, the weights and neurons from the input layer are multiplied, shifted by a bias and then activated by the activation function. To compute the values in the output layer, you may choose not to have a bias and have an identity activation function on this layer, so that this last calculation is just "scaling".

Is it standard to have a "scaling" layer? You could say that there is a bias associated with each neuron, except those in the input layer correct (and those in the output layer when it is a scaling layer)? Although I suppose you could immediately shift any value you're given. Does the input layer have a bias?

I have seen bias represented as an extra unchanging neuron in each layer (except the last) having value 1, so that the weights associated with the connections from this neuron correspond to the biases of the neurons in the next layer. Is this the standard way of viewing bias? Or is there some other way to interpret what bias is that is more closely described by "a number that is added to the weighted sum before activation"?

Answer 1

The purpose of the input layer is just to conceptually represent the input and, in case it is necessary, define the dimensions of the input that the neural network expects. In fact, some neural networks, such as multi-layer perceptrons, expect a fixed-size input, but not all of them: fully convolutional networks can deal with inputs of different dimensions.

The input layer doesn't contain neurons (although in the diagrams that you will come across they are usually represented as circles, like the neurons, and that's probably why you are confused!), so it also does not contain biases, linear transformations, and non-linearities. In fact, in the context of neural networks, you could define a neuron as some unit/entity that performs a linear or non-linear transformation (to which you can add a bias). Note that the hidden and output layers can contain biases because they contain neurons that perform a linear or non-linear transformation.

However, although I have never seen it (or I don't recall having seen it), I would not exclude the existence of an input layer that transforms or augments the inputs before passing them to the next layer. For example, one could implement a neural network that first scales the input to a certain range, and the input layer could do this, although, in practice, this is typically done by some object/class that does not belong to the neural network (e.g. tf.data.Dataset).