I'm working on an implementation of the backpropagation algorithm for a simple neural network, which predicts a probability of survival (1 or 0).
However, I can't get it above 80%, no matter how much I try to set the right hyperparameters. I suspect that's because my backpropagation is implemented incorrectly, since I tried 2 different types of code and both give me the same results.
Is there a way to determine whether my implementation of backpropagation is correct?
Don't feel too bad for having gotten it slightly wrong because backpropagation is notoriously difficult to implement [1].
There is a technique called gradient checking, which you can implement to test the correctness of your backpropagation implementation. I would argue that even gradient checking is a little tricky to implement.
How does gradient checking work?
Backpropagation computes the gradients $\frac{\partial J}{\partial \theta}$, where $\theta$ denotes the parameters of the model. $J$ is computed using forward propagation and your loss function.
But because forward propagation is fairly straightforward to implement, most people are usually confident that you got its implementation correct. So, the trick is to use the value of $J$ to verify your code for computing $\frac{\partial J}{\partial \theta}$.
We know that by definition, the gradient or derivative is given by: $$\frac{\partial J}{\partial \theta} = \lim_{\epsilon \rightarrow 0} \frac{J(\theta + \epsilon) − J(\theta − \epsilon)}{2 \epsilon}$$ Since we trust our calculation of $J$, we can easily compute the value of $J(\theta + \epsilon) − J(\theta − \epsilon)$.
For more information, see this Andrew Ng's video lecture and these notes.
For future reference, you can check your correctness by the finite difference method.
http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf (p.23)
The given so far answers focus on numerical methods to check your gradients. It is really useful, especially if one doesn't have much experience in backprop.
But I'd like to add here a pure practical "sanity check", relatively fast and easy to perform, which also works for other issues, e.g. (rough) hyperparameters selection. To see if your network makes sense, reduce the training set to a few examples and try to overfit the network. If the loss falls to zero and training accuracy skyrockets to 1, it means that both passes work correctly and can move on to the real training. Otherwise, something's not right and should dive into specific parts of the network, in particular check the gradients numerically.
