How does a sigmoid neuron output 1 with 0 as input?

Question

Sorry if that is a dumb question. I just started to learn about machine learning.

I'm reading this book about neural networks: http://neuralnetworksanddeeplearning.com/chap1.html#a_simple_network_to_classify_handwritten_digits

It explains how an artificial neural network classifies handwritten digits applying weights and biases to inputs and each input is assigned a value between 0 and 1, with 0 being white pixels and 1 black pixels.

Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1, which means that all neurons in output layer will output 1. This is not the intended behavior.

What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input? Is this done by assigning a negative weight or using a different activation function?

Answer 1

Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1

Not necessarily. In a neural network, there are weights $W$, synapses if you will, connecting layers. These weights are randomly initialised, and can be negative. Furthermore, assuming a fully connected network, each hidden activation receives the activations from all neurons in the previous layer as input. A mixture of randomly negatively and positively scaled inputs, summed and passed through a non-linearity, could lead to either a positive or negative output. Hence why an input image of black pixels (0) will not cause all the hidden neurons to also be black (0) or white (1). This of course counts for all subsequent layers too.

What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input?

As @Minh-Long Luu points out, you probably don't want or need a neural network to do this... In fact, I'm not even sure if this is possible (assuming training via backprop) because it sounds like you're after a heaviside step-function for the non-linear activation, which would be non-continuous and therefore non-differentiable, making it impossible to train the NN with stochastic gradient descent. Perhaps a flipped sigmoid centered at 0.5 would be more appropriate?

Answer 2

This inverter is an interesting initial application of an artificial neuron.

Yes, you need a negative weight and... no, the sigmoid suffices if you do with a close enough approximation.

See how an unrestricted optimization could progress from a very inexact approximation to the Heaviside step in the limit, flipped and shifted as Giulio suggests.

If you want to test other numbers, here is the WolframAlpha link.