I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between stochastic/minibatch gradient descent and batch gradient descent:

A comparison between the prediction results

The training data is a collection of point coordinates (x,y). The labels are 0s or 1s (below or above the parabola). enter image description here

As a test, I am doing a classification task. My objective is to make the NN learn which points are above the parabola (yellow) and which points are below the parabola (purple).

Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb

  • Why is the batch gradient descent performing so poorly with respect to the other two methods?
  • Is it a bug? But how can it be since the code is almost identical to the
    minibatch gradient descent?
  • I am using the same (randomly chosen with try and error) hyperparameters for all three
    neural networks. Does batch gradient descent need a more accurate
    technique to find the correct hyperparameters? If yes, why so?

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.

The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.

You can check this out for more on the differences between the three. Hope this helped!

  • 1
    $\begingroup$ Thanks, I am understanding this better. So, basically, BGD is not very used because I guess most loss functions have multiple minimas. Am I wrong? What other loss functions could I use in this particular example? $\endgroup$ – Pigna Sep 27 '19 at 12:15
  • 1
    $\begingroup$ Yes. What other loss functions could I use in this particular example - You have used the difference as loss (written as 'error' in your code) if I'm not wrong? If so, you can also try out Magnitude of the difference, square of the difference, logarithmic cross-entropy - looking up common loss functions will give you an abundant number of potential ones. $\endgroup$ – ashenoy Sep 27 '19 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.