I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how they are used by a neural network to fit a line to a set of data.

I understand the purpose of an activation function, but what I'm struggling with is how we go from deciding which type of activation function to use for each neuron to actually fitting a line to our data.

In an attempt to illustrate this question, here is a YouTube video which demonstrates how a fitted line would change as data is passed through a neural network using various types of activation functions.

At the moment, it just seems like magic that we pass data through and depending on which type of activation function being used the line somehow changes in various ways and I have no idea why it's doing what it's doing.

Any help with this or guidance regarding what knowledge gaps I might have that are preventing me from understanding this would be greatly appreciated! My goal in all in asking this is to be able to do the math by hand in a much smaller network to demonstrate that I understand what's happening behind the scenes.


1 Answer 1


Most of the commonly used activation functions for neural networks work reasonably well on most machine learning tasks. You do need to care about the function used in the output layer though.

For hidden layers, the activation functions do change the possible shapes that the final output can make easily. The basic shape of each neuron's output is the activation function shifted left or right, and in the next layer those shapes can be combined in any multiple (including negative weights that invert the shape vertically). However, once you have a few dozen neurons and 2 or more layers, the differences between what you can make are subtle and difficult to predict. Whilst it is possible to tune a neural network by trying different activation functions in hidden layers, it is more common to pick one that works well enough - some variation of ReLU is a reasonable choice, especially for deeper networks.

The output layer is an exception. You want the output to match the range of your training data, and ideally naturally represent the output type for your problem.

There are three very common choices for output layer activation function:

  • For regression problems, do not use any activation function in the output - sometimes this is called the "linear" activation function.
  • For single classifiers or arrays of non- exclusive classifiers, use sigmoid activation to represent the probability of being in the class(es)
  • For multi-class exclusive classifiers, use softmax activation which naturally sums to 1.0 over all the outputs and represents probability of being in each class.

In all cases, the activation function does not affect the how of fitting the output to the data. That is done by calculating the gradient of all the parameters in the neural network with respect to a cost function (which measures how close the outputs were to the data) and taking a small step in the opposite direction to the gradient, which should reduce the value calculated by the cost function. This is repeated many times - measuring the gradient again and taking a new step in a different direction each time. This is gradient descent. Calculating the gradients is done using back propagation, which is based on the chain rule for differentiating combined functions. The activation function is one of the functions being combined, so is the sum of outputs from the previous layer.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .