4
$\begingroup$

I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between Stochastic/Mini-Batch Gradient Descent and Batch Gradient Descent:

A comparison between the prediction results

The training data is a collection of point coordinates (x,y). The labels are 0s or 1s (below or above the parabola). enter image description here

As a test, I am doing a classification task. My objective is to make the NN learn which points are above the parabola(yellow) and which points are below the parabola(purple).

Here is the link to the notebook: https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb

  • Why is the Batch Gradient Descent performing so poorly with respect to the other two methods?
  • Is it a bug? But how can it be since the code is almost identical to the
    Mini-Batch Gradient Descent?
  • I am using the same (randomly chosen with try and error) hyperparameters for all three
    neural networks. Does Batch Gradient Descent need a more accurate
    technique to find the correct hyperparameters? If yes, why so?
Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
2
$\begingroup$

Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.

The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.

You can check this out for more on the differences between the three. Hope this helped!

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks, I am understanding this better. So, basically, BGD is not very used because I guess most loss functions have multiple minimas. Am I wrong? What other loss functions could I use in this particular example? $\endgroup$
    – Pigna
    Commented Sep 27, 2019 at 12:15
  • 1
    $\begingroup$ Yes. What other loss functions could I use in this particular example - You have used the difference as loss (written as 'error' in your code) if I'm not wrong? If so, you can also try out Magnitude of the difference, square of the difference, logarithmic cross-entropy - looking up common loss functions will give you an abundant number of potential ones. $\endgroup$
    – ashenoy
    Commented Sep 27, 2019 at 14:32
0
$\begingroup$

Because the computation of Batch Gradient Descent(BGD) is more stable(less fluctuated) than Mini-Batch Gradient Descent(MBGD) and Stochastic Gradient Descent(SGD), so BGD less easily(more difficultly) escapes local minima than MBGD and SGD, resulting in poor learning even thouch BGD can get more accurate output than MBGD and SGD.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .