Why is a bias parameter needed in neural networks?

Question

I have read several resources, including previously asked questions such as this. I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can perform feature transformation, what is the need of a bias term?

Since the weights are learnt, my network can optimise to fit the data. For example, if my data is in 2D coordinate plane, my equation without bias for a perceptron for the layer will be $W_1X_1 + W_2X_2$ where $X_1$ is x coordinate and $X_2$ is y coordinate, making $W_1$ and $W_2$ coefficients of all vectors along x and y direction. Their linear combination will cover the whole plane which allows my data to be transformed across a line with 0 intercept.

For example, if my weight is 1.0 for input x, and my bias is 0.1, I might as well have weight $1+(0.1/\bar x)$ (or any other value descriptive of x) and 0 bias to get the same result.

Similar things happen for the arguments related to activation mentioned in the marked solution to the referenced question.

In such a scenario, why is the bias needed?

Edit: A lot of the answers offer reasonable arguments for the perceptron/single layer case, but perceptron was just an example. Do they hold for deep neural networks as well, because that allows for previous layers better transformation of inputs? As mentioned by some, 0 input will truly cause a problem which I agree with.

Answer 1

It's not strictly "needed." In fact, if you look at things like Keras, you will see that layers have a use_bias parameter, which defaults to True, but you can set to False, of course.

For an intuition about why bias is useful rather than required, consider a simple $\mathbb{R}^2$ example. Imagine that we have some data that we are attempting to fit a straight line to.

When generating our line, we can iteratively update the slope using something like gradient descent and completely ignore the y-intercept, or bias term. In the end we will find a line, centered at the origin, that has the identical slope to a best-fit line that passes through the data.

If you have this mental picture, take it a step further. Using the bias term (y-intercept), we can then adjust that line up or down (bias the line up or down) by whatever amount is needed to minimize the loss.

If you think about it, for any line $y=mx+b$, we could think of a specific line $y=4x$ as representing the fundamental line for all lines with that slope. Really, they are all the same line that we can slide up or down the y-axis to place them where we need them to be.

Coming back to training a neural network, the bias, therefore, is not required, but can be very useful in allowing us to adjust the output of a neuron up or down as required to better fit the data, possibly easing the difficulty of training subsequent layers/neurons.

Answer 2

Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class y=0 and the second blob will have the class y=1.

import numpy as np
x = np.concatenate([
  np.random.normal(loc=0, scale=1, size=2*1000).reshape(-1,2),
  np.random.normal(loc=3, scale=1, size=2*1000).reshape(-1,2)
])
y = np.concatenate([np.zeros(1000), np.ones(1000)])

We can plot it with something like scatter(x[:,0], x[:,1], c=y):

I'll use torch, so I convert these to torch tensors and use its data wrangling classes to shuffle and split into batches.

import torch
x, y = torch.tensor(x).float() , torch.tensor(y).long()
dataset = torch.utils.data.TensorDataset(x,y)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)

Now let's make two neural networks - just a linear layer with 2D inputs and two outputs for each class. The only difference is that the first one will have bias=False and the second one bias=True.

model1 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=False))
model2 = torch.nn.Sequential(torch.nn.Linear(2, 2, bias=True ))

I assume that our networks return logits of the classes - so cross-entropy as a loss function. I've hacked together this pretty standard code that, given a model and an optimizer goes through one epoch and returns average loss:

loss_fn = torch.nn.CrossEntropyLoss()
def optimize_epoch(model, optimizer):
    total_loss = 0
    for n, (inputs, labels) in enumerate(dataloader):
        optimizer.zero_grad()
        outputs = model.forward(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    return total_loss / n

Running this optimization for 50 epochs and collecting loss history:

optimizer1 = torch.optim.SGD(model1.parameters(), lr=0.01)
optimizer2 = torch.optim.SGD(model2.parameters(), lr=0.01)

losses1, losses2 = [] , []
for _ in range(50):
    losses1.append(optimize_epoch(model1, optimizer1))
    losses2.append(optimize_epoch(model2, optimizer2))

plot(losses1, label="model1"); plot(losses2, label="model2"); legend()

You'll see a striking difference in performance between the two models:

---

As per @Stef request, here are the scatterplots for predicted classes for each model. Obtainable via scatter(x[:,0], x[:,1], c=model1.forward(x).argmax(axis=1).

Here you can clearly see that the separating line for bias=False goes through (0,0) in accordance with most of other answers here.

Answer 3

No matter what you make $W_1$ and $W_2$, if $X_1$ is 0 and $X_2$ is 0 then $W_1X_1+W_2X_2$ is 0 which (in a typical classification application) means the classifier is completely unsure which class it belongs to.

Additionally, mirroring a point across the origin (that is, assigning $X_1 \leftarrow -X_1$ and $X_2 \leftarrow -X_2$) will also negate the output. The classifier is only able to provide classifications where one class is mirrored across the origin from the other class.

Adding a bias term - $W_1X_1+W_2X_2+W_3$ - solves these problems. The classifier can draw any line to separate the two classes, not only lines that pass through the origin (0,0).

Answer 4

If you have data generated from $y = 5\,x + 3$

How do you expect the simplest neural net $y = w_1 x$ to adjust the data ?

That is why $b$ is useful.

Answer 5

Let's interpret each node of a layer as a transformation of sub-feature-inputs into a certainty value for the presence (or absence) of a feature.

Then, for example a dense-layer first looks at each of the sub-features independently. It classifies for each sub-feature whether it counts as evidence of presence or absence of the feature (which the node wants to detect) which is expressed by the sign of the weight of edge of the sub-feature to the node. Then the absolute weight of the edge tells us how strong of an evidence the corresponding sub-feature is.

Now, for some features it might be much easier to detect evidence for their absence, than to find evidence of their presence. In such cases, the easiest way for the node to learn the feature might be to just find lots of measures which are evidence for the absence of the feature it wants to detect, and then use the bias as a counter-term which decides whether the evidence for absence found by the sub-features is sufficient to rule out the existence of the feature.