I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between stochastic/minibatch gradient descent and batch gradient descent:
As a test, I am doing a classification task. My objective is to make the NN learn which points are above the parabola (yellow) and which points are below the parabola (purple).
Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb
- Why is the batch gradient descent performing so poorly with respect to the other two methods?
- Is it a bug? But how can it be since the code is almost identical to the
minibatch gradient descent?
- I am using the same (randomly chosen with try and error) hyperparameters for all three
neural networks. Does batch gradient descent need a more accurate
technique to find the correct hyperparameters? If yes, why so?