I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient descent (so all 155 samples) or use mini batch/stochastic gradient descent. I have read that using smaller mini batch sizes allows better generalisation, but as the batch size is very very small computationally it shouldn't be a burden to use BGD.

  • $\begingroup$ Input size of 155x155 is rather ambiguous i feel, it could mean a input with dimension 155x155 and not the size of the training set. If the number of samples is small as in your case, I believe that a BGD should converge faster than a SGD. However, in your case where you have so few samples, the time to converge shouldn't be that much significant. FYI: Some Mini batch sizes range up to 256 samples $\endgroup$
    – calveeen
    Feb 19 at 15:15
  • $\begingroup$ Thanks for your answer! To clarify, the training set size is 155x155 not the dimension. $\endgroup$
    – Max
    Feb 19 at 15:17
  • $\begingroup$ You mean (n_samples, n_features) = (155, 155) right? $\endgroup$
    – calveeen
    Feb 19 at 15:18
  • $\begingroup$ Yes, exactly right! $\endgroup$
    – Max
    Feb 19 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.