10
$\begingroup$

I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python.

Here's one of the things I'm wondering about: I'm training the neural network on sample data, and I've got 1,000 samples. The training consists of gradually changing the weights and biases to make the cost function result in a smaller cost.

My question: Should I be changing the weights/biases on every single sample before moving on to the next sample, or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
11
$\begingroup$

Should I be changing the weights/biases on every single sample before moving on to the next sample,

You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the dataset before working through it each time.

or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

You can do this, it is called batch gradient descent, or in some cases (especially in older resources) just assumed as the normal approach and called gradient descent.

Each approach offers advantages and disadvantages. In general:

  • SGD makes each update sooner in terms of amount of data that has been processed. So you may need less epochs before converging on reasonable values.

  • SGD does more processing per sample (because it updates more frequently), so is also slower in the sense that it will take longer to process each sample.

  • SGD can take less advantage of parallelisation, as the update steps mean you have to run each data item serially (as the weights have changed and error/gradient results are calculated for a specific set of weights).

  • SGD individual steps make typically only very rough guesses at the correct gradients to change weights in. This is both a disadvantage (performance of the NN against the objective on the training set can decrease as well as increase) and an advantage (there is less likelihood of getting stuck in a local stationary point due the "jitter" these random differences cause).

What happens in practice is that most software allows you to compromise between batch processing and single-sample processing, to try and get the best performance and update characteristics. This is called mini-batch processing, which involves:

  • Shuffling the dataset at the start of each epoch.

  • Working through the shuffled data, N items per time where N might vary from maybe 10 to 1000, depending on the problem and any constraints on the hardware. A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.

  • Calculate the update required for each small batch, then apply it.

This is nowadays the most common update method that most neural network libraries assume, and they almost universally will accept a batch size parameter in the training API. Most of the libraries will still call simple optimisers that do that SGD; technically it is true, the gradients calculated are still somewhat randomised due to not using the full batch, but you may find this called mini-batch gradient descent in some older papers.

Improve this answer
$\endgroup$
5
  • $\begingroup$ 'A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.' How do you determine this? I haven't seen any resource where it can be commented when the batch size is just enough for peak parallelization $\endgroup$
    – user9947
    Commented May 25, 2019 at 6:11
  • 1
    $\begingroup$ @DuttaA e.g. stackoverflow.com/questions/46654424/… and pugetsystems.com/labs/hpc/… $\endgroup$
    – Neil Slater
    Commented May 25, 2019 at 6:40
  • $\begingroup$ "A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel." -- You have it backwards! The first heuristic is to process the smallest mini-batch size that results in acceptable performance. With many models, however, you hit memory limits before you saturate hardware efficiency, so you end up running the largest model that will fit into RAM. Generally, though, smaller batch sizes find better minima since they exhibit more stochasticity. A caveat is that batch-norm breaks with very small batch sizes. $\endgroup$
    – Aleksandr Dubinsky
    Commented May 25, 2019 at 16:27
  • $\begingroup$ @AleksandrDubinsky RAM is hardware. $\endgroup$
    – user9947
    Commented May 26, 2019 at 5:03
  • $\begingroup$ @AleksandrDubinsky: Yes, but I don't want to import so much detail and caveats into the answer on that topic. I'm not 100% sure of the heuristic you suggest being the most used/popular; there are 2 performance metrics (training time and cross-validation results), and more stochasticity is not necessarily a good thing for best generalised metrics. As you note, adding in batch norm (and maybe other things) makes the landscape even more complex, and a bit much for this answer to cover. Many people do seem to end up hitting GPU RAM limits with low training time being the primary goal. $\endgroup$
    – Neil Slater
    Commented May 26, 2019 at 7:41
1
$\begingroup$

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical and very slow.

This is where, two optimization techniques became prominent.

  1. Mini-Batch Gradient Descent
  2. Stochastic Gradient Descent (SGD)

In mini-batch gradient descent, you use a batch size that is considerably less than total no. of training examples and update your weights after passing through these examples.

In stochastic gradient descent, you update the weights after passing through each training example.

Coming to advantages and disadvantages of the three methods we discussed.

  • Batch gradient descent gradually converges to the global minimum but it is slow and requires huge computing power.

  • Stochastic gradient descent converges fast but not to the global minimum, it converges somewhere near to the global minimum and hovers around that point, but doesn't converge ever to the global minimum. But, the converged point in Stochastic gradient descent is good enough for all practical purposes.

  • Mini-Batch gradient is a trade-off the above two methods. But, if you have a vectorized implementation of the weights updation and you
    are training with a multi-core setup or submitting the training to
    multiple machines, this is the best method both in terms of time for training and convergence to global minimum.

You can plot the cost function, w.r.t the no. of iterations to understand the difference between convergence in all the 3 types of gradient descent.

  • Batch gradient descent plot falls smoothly and slowly and gets stabilized and gets to global minimum.

  • Stochastic gradient descent plot will have oscillations, will fall fast but hovers around global minimum.

These are some blogs where there is detailed explanation of advantages, disadvantages of each method and also graphs of how cost function changes for all the three methods with iterations.

https://adventuresinmachinelearning.com/stochastic-gradient-descent/

https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged .