How does an activation function's derivative measure error rate in a neural network?

Question

A blog post called "Text Classification using Neural Networks" states that the derivative of the output of a sigmoid function is used to measure error rates.

What is the rationale for this?

I thought the derivative of a sigmoid function output is just the slope of the sigmoid line at a specific point.

Meaning it's steepest when sigmoid output is 0.5 (occuring when the sigmoid function input is 0).

Why does a sigmoid function input of 0 imply error (if i understand correctly)?

Source: https://machinelearnings.co/text-classification-using-neural-networks-f5cd7b8765c6

We use a sigmoid function to normalize values and its derivative to measure the error rate. Iterating and adjusting until our error rate is acceptably low.

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

def train(...)
    ...
    layer_2_error = y - layer_2
    layer_2_delta = layer_2_error * sigmoid_output_to_derivative(layer_2)
    ...

UPDATE

Apologies. I don't think I was clear (I've updated the title)

I understand we don't need to use sigmoid as the activation funtion (we could use relu, tanh or softmax).

My question is about using the derivative to measure the error rate (full quotation from article above in yellow) -> what does the derivative of the activation function have to do with measuring/fixing the "error rate"?

Answer 1

This derivative is used when calculating the error of your machine learning algorithm during gradient based minimization methods. Read below for more info.

When performing supervised classification (with X, Y data vectors of inputs and outcome data to train with) you begin with the error function

E(X, Y; θ)= ∑_i (ƒ(x_i; θ)-y_i)²

for total error over all data instances i, where f is your neural network, linear regression,...method of interest and θ is the set of weights. The goal here is to find weights that minimize your error when predicting training data (y) (which ideally generalizes to new data as well). To be explicit, ƒ(x_i; θ); outputs value of interest which should be y_i. And E measures how far off it is in prediction.

So to train your classifier, you optimize E with something like gradient descent. Thus when ∂E/∂θ = 0 (for a particular θ), that means you hit a local minimum for the error function, or a point where the error in the current state of the predictor is low, meaning it is (hopefully) a good predictor.

Note the ƒ here is not the same as an activation function, as a neural network is defined differently than in linear regression, etc. and must perform a special kind of gradient descent called backpropagation.

So when you take ∂E/∂θ, what does it equal for a neural net? You should note the activation functions derivative is involved which is how it’s used to measure error so to say.

Answer 2

You dont need a sigmoid function if you dont want one. Any differentiable function will do. Sigmoid functions are just one of many suitable functions. You could write your own differentiable function if you want a propriety solution.

Answer 3

Measuring the error rate of a neural network does not involve the derivative of the sigmoid function at all. It only needs the neural networks outputs, and the expected outputs. It does not matter how the neural network got to those outputs, the outputs only have to be based on the inputs. What the other of that specific text is trying to say is that the derivative of the sigmoid function is important when you use the algorithm back propagation to train the neural network. This method involves using derivatives to optimize the neural network. While this technique is more complicated with neural networks because of how many variables are involved, you can easily see the basics of this approach if you look at a calculus textbook and look at the chapter that is usually titled Applications of the derivative.

Answer 4

The derivative of your loss is the "slope" at that given prediction. So by "moving down along the slope", we can reduce the value of the loss function.

Intuitively, one can imagine a ball rolling down a slope in the direction of the tangent. Eventually the ball will roll towards the lowest point (global minima), or a small pothole (local minima), or even some weird looking shape (saddle point) if you are very unlucky.

With some fixed input/output pair, we can consider the loss function as a value of the parameters (weights). Here, the Z axis is the value of your loss f (what you are trying to optimize), while the X,Y axis are the parameters of the loss f (what you are allowed change):

Note that the image provided is a bit misleading. In practice, we only know the value of the loss function at the location we have probed. The entire graph can only be drawn if we evaluate the loss function for every parameter. So we can't just look at the picture and know which location is the best.

Recall that the goal of the prediction task is to update the parameters in a way such that we minimize the loss function for some given input/output pair. Assuming the loss function is reasonably smooth, then our best bet of reaching a lower value of the error rate is by taking a small step towards the direction of the slope, which is the derivative of the function. The size of the "small step" is called the learning rate.

Under the assumption that the loss function looks similar given different input/output pairs, by iteratively moving down the slope for each datapoint, we can eventually reach a "good" set of weights that give a low error.

This is the motivation and idea of the back-propagation algorithm.

A good follow-up question might be: Why do we say that the loss function is similar given different data points? This is basically the same as assuming that there is some regularity in the data such that there is a "most normal looking loss function" that the data-set naturally approaches.

Hopefully that helps!

Answer 5

In order to train a neural network, you have to adjust each weights and biases to reduce the cost to as minimum as possible. The only way to do so is to subtract a small amount of the partial derivative of cost w.r.t. w, b from the respective parameters.

if J is our cost function, after each iteration:

w = w - lr*dJ_dw,      //where lr is a small scalar called learning rate and dJ_dw is the partial derivative of cost function w.r.t. w

and same for bias

b = b - lr*dJ_db,    //dJ_db is the partial derivative of cost function w.r.t b

let's look at how the partial derivatives are calculated.

using sigmoid as activation function, and squared error function as cost, we have:

z = w*x + b
a = sigmoid(z)       // sigmoid(z) = 1.0 / (1.0 + exp(-z)), a is the final output

J = (a - y)*(a - y)  // where y is the expected output

For us to calculate partial derivatives of this cost function w.r.t. w, b, we need to use, chain rule of derivatives as:

dJ_dw = dJ_da * da_dz * dz_dw    // dJ_da is the partial derivative of cost w.r.t. activation, a
dJ_db = dJ_da * da_dz * dz_db

in the above equations, da_dz is the derivative of activation function (sigmoid in our case) which is sigmoid(z).(1 - sigmoid(z))