Is the bias also a "weight" in a neural network?

Question

I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "bias".

$$ Y= \sigma(\text{weight} * \text{input})+ \text{bias} $$

However, all the sources I've found when you train the network you just adjust the weights. Not the bias.

However, they never mention what the bias should do, which leads me to think that you just merge all weights and biases in a $W$ vector and call it weights, even though there are also biases. Is that correctly understood?

Answer 1

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias.

One notable text using this convention is Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz and Shai Ben-David. In section 20.1 there is a diagram of a neural network where a neuron outputting a constant value is added to each layer which you might find helpful.

To understand why this works, suppose the outputs of the previous layer are $u_1, \dots, u_n, u_{n + 1}$, where $u_{n + 1}$ is always $1$. Then a neuron in the next layer (without a bias) computes

$$ \sigma\left(\sum_{i = 1}^{n + 1} w_i u_i \right) = \sigma\left(\sum_{i = 1}^n w_i u_i + w_{n + 1}\right),$$ where $\sigma$ is the activation function. So, the weight $w_{n + 1}$ just serves as the bias because it is multiplied by $u_{n + 1} = 1$.