I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "bias".

$$ Y= \sigma(\text{weight} * \text{input})+ \text{bias} $$

However, all the sources I've found when you train the network you just adjust the weights. Not the bias.

enter image description here

However, they never mention what the bias should do, which leads me to think that you just merge all weights and biases in a $W$ vector and call it weights, even though there are also biases. Is that correctly understood?

  • 1
    $\begingroup$ bias is just a simple kind of weight to shift the separation line, it can be imagined as a weight which is connected to input (original or previous layer) of value 1 always $\endgroup$ – datdinhquoc Apr 14 at 8:39

Yes, it is not unusual to omit the bias by adding a neuron which always outputs a constant 1, which will then be multiplied by an appropriate weight to give the same formula as you would get using an explicit bias.

One notable text using this convention is Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz and Shai Ben-David. In section 20.1 there is a diagram of a neural network where a neuron outputting a constant value is added to each layer which you might find helpful.

To understand why this works, suppose the outputs of the previous layer are $u_1, \dots, u_n, u_{n + 1}$, where $u_{n + 1}$ is always $1$. Then a neuron in the next layer (without a bias) computes

$$ \sigma\left(\sum_{i = 1}^{n + 1} w_i u_i \right) = \sigma\left(\sum_{i = 1}^n w_i u_i + w_{n + 1}\right),$$ where $\sigma$ is the activation function. So, the weight $w_{n + 1}$ just serves as the bias because it is multiplied by $u_{n + 1} = 1$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.