Bias equal to 1 and neuron output equal to -1 in neural networks

Question

I have read that bias in neural networks is used to avoid situation in which output of neuron is equal to 0. But what if the same output is equal to -1 and we add 1 to it? Isn't it the same issue as in case of zero output and no bias?

Answer 1

Short answer: The bias in a neural network is not used to avoid an output of 0. The bias is used to shift the activation function.

What this means

An activation I'm sure almost anyone is familiar with, the sigmoid function:

$$f(x)=\frac{1}{1+e^{-x}}$$ Which looks like:

If we use the sigmoid function as the activation of all neurons (without bias'), if a value needs to be shifted, we can never do this directly. Instead, it can only ever be scaled in such a way that it shifts on the next layer when we multiply by more weights and add. That might be a little confusing, so imagine it like this:

If you had a desired output of 2, how would you ever reach that with the sigmoid function? The answer is you can't, and that's where a bias comes in. With a bias of 1.5 and an output of 0.5, you could reach your desired output of 2.

For an even more concrete example, let's take a look at the Rectified Linear Unit (ReLU):

$$ f(x) = max(0, x)$$

Which looks like:

Now the goal of a neural network is to approximate a function, so lets in this case try and approximate a 2d function using the ReLU activation function. Conveniently, there's a really nice example in desmos of exactly this. As you can see, each ReLU equation employs a bias to move the function around. I challenge you to try and fit $f\left(x\right)=x^{3}+x^{2}-x-1$ without using a bias, and you'll really quickly see the need for a bias:

*Note I should've included the sign along with the numeric value of each bias inside the red bubble below

ETA: Variance!!!

How could I forget.

An important distinction to make is this - An output of 0 is perfectly normal in a neural network. As an example, look at ReLU! A proven, extremely effective activation function that for half of all inputs, outputs a 0!

The issue, is when 0 decreases your variance. It's extremely important to maintain variance in a neural network. By this I mean: The spread of the inputs should roughly equal the spread of the outputs. The reason for this is obvious once you know:

• If spread gradually increases, the values in the network will explode to infinity, rendering most activation functions useless and slowing down calculations by orders of magnitude
• If spread gradually decreases, the values in the network will vanish to 0, meaning the network doesn't calculate anything at all!

So how does this tie in? Well, sticking with the ReLU example, say you have 9 positive inputs and 1 negative input. When that negative input goes through the ReLU function, it will come out as 0. This will invariably decrease variance in the long run (there are some cases where it increases initially, but eventually it will decrease), eventually vanishing all values to 0. But with a bias, this can be counteracted. By keeping it even slightly away from 0, in combination with the parameters following, you can ensure that the variance stays roughly the same. Alas, the need for a bias.

For more information on keeping variance level, see my question from when I was just learning to count, and how I came to know exploding numbers all too well...