Why is batch gradient descent performing worse than stochastic and minibatch gradient descent?

Question

I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between stochastic/minibatch gradient descent and batch gradient descent:

The training data is a collection of point coordinates (x,y). The labels are 0s or 1s (below or above the parabola).

As a test, I am doing a classification task. My objective is to make the NN learn which points are above the parabola (yellow) and which points are below the parabola (purple).

Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb

• Why is the batch gradient descent performing so poorly with respect to the other two methods?
• Is it a bug? But how can it be since the code is almost identical to the
  minibatch gradient descent?
• I am using the same (randomly chosen with try and error) hyperparameters for all three
  neural networks. Does batch gradient descent need a more accurate
  technique to find the correct hyperparameters? If yes, why so?

Answer 1

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.

The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.

You can check this out for more on the differences between the three. Hope this helped!