I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of inputs and a single output.

calculating errors and updating weights The first line gives the error that will be used to update the weights going from the hidden layer to the output layer. $t$ represents the target output, $a$ represents the activation and the formula is using the derivative for the sigmoid function $(a(1-a))$. Then, the weights are updated with the learning rate, multiplied by the error and the activation of the given neuron which uses the weight $w_h$. Then, the next step is moving on to calculate the error with respect to the input going into the hidden layer from the input layer (sigmoid is the activation function on both the hidden and the output layer for this purpose). So we have the total error * derivative of the activation for the hidden layer * the weight for the hidden layer.

I am following this train of thought as it was provided, but my question is — if the activation is changed to $tanh$ for example and the derivative of $tanh$ is $1-f(x)^2$, then would we have the error formula update to $(t-a)*(1-a^2)$ where $a$ represents the activation function so $1-a^2$ is the derivative of $tanh$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.