I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The result is that when I use the A-values, the neural network learns faster and more stably than with the calculation using the Z-values. The calculation should be incorrect. So why does it work better, yielding better results and a more stable outcome?

Z=x×w+b A=activationfunction(Z)

Wrong calculation with better results: dA/dZ=activationfunction_derivative(A)

delta = deltas[-1].dot(self.weights[i].T) * self.leaky_relu_derivative(self.activations[i])

Right calculations but with worse results:




You must log in to answer this question.