I just want to verify the my intuition of why activation functions are necessary. For this example lets consider a network that classifies numbers 0-9. A network WITHOUT an activation function will be able to do well on numbers that are similar to the sizes of the numbers in the training set but it will struggle if numbers appear to be darker or lighter because it is linear and cannot take both size and lightness/darkness into account. In a neural network completeing the same task but WITH activation functions will be able to take into account both orientation and lightness/darkness because the weights will learn all possible relationships between the pixel values in the data set and the sigmoid(or other acctivation function) will then take number that are slighty lighter or darker and transform/smooth them so that the network could ouput the same probability as if it were the nomral darkness/lightness. Does this intuition sound about right or is it incorrect in some way ?

Here is another example of what I am trying to descirbe if this makes more sense: I understand it is a non linearity but mabey let me try and explain what I am thinking in a diffrent way. In a network that is detecting numbers 0-9 the network has learned for example all the relationships between the pixels of a 7 of standard darkness in all sorts of positions. Then during test time a lighter 7 comes along with slighlty lighter pixel values. In one of the neurons in the input layer after the lighter 7’s input has been multiplied by the weights and biases for that neuron when it is transformed by the sigmoid function it will have aproximatly the same value the darker one after passing through sigmoid in neuron one would be 0.994 and the lighter one 0.992 so then the lighter one will get treated the same through all later layer of the network and get classified correclty. Does that make sense ?

  • $\begingroup$ Why do you claim "it will have aproximatly the same value the darker one after passing through sigmoid in neuron one would be 0.994 and the lighter one 0.992"? Imagine you simply have Relu activation function, it's obviously not the case but the output of the whole network can still work well. $\endgroup$
    – cinch
    Feb 16 at 4:54
  • $\begingroup$ Can you please put your specific question in the title? "Activation function intuition question" is just the announcement of the question, but we already know that you have a question. $\endgroup$
    – nbro
    Feb 16 at 10:46

1 Answer 1


Activations can help with brightness, but not in the way you described, not by smoothing.

Without activation, all your layers collapse to a single linear operation, just because in matrix operations W1*W2 = W3. That means the prediction from every pixel is independent. Like, you can learn that the pixel at position (5,6) is brighter for #7 and darker for #1. That's it; it's the only level of complexity you have.

With activations, you can build a deeper network with more complex relationships. It can train that neuron 17 on layer 2 could have average brightness and then it can be subtracted from all other pixels.

As for size change, activations don't help with it. In fact, even convolutional layers are not scale-invariant, let alone multiple linear layers. For that, you would need something like a U-Net or multi-scale networks.

Bottom line, people learned that non-linearity is extremely important, but the exact shape of it is not so much.

Google has a nice playground for MLP https://playground.tensorflow.org/ And an extended explanation, including MNIST https://cloud.google.com/blog/products/ai-machine-learning/understanding-neural-networks-with-tensorflow-playground

  • $\begingroup$ Thanks of the response. I understand that the activation function does not help with the shape. If the two 7's are the same in every way except brightness/darkness when those pre activation values are entered into a sigmoid they will come out to be around the same 0.993 vs 0.991 so then they will be treated the same throughout the rest of the network. Does that make sense ? $\endgroup$
    – Stef
    Feb 15 at 14:46
  • 1
    $\begingroup$ The case you describing is when nn is very confident anyway. Now imagine it gave probability not 99.3%, but let's say 90%. With half the brightness it will recalculates as sigmoid(logit(p/2)) = 73%, which is almost triple error. $\endgroup$ Feb 19 at 11:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .