I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python.

Here's one of the things I'm wondering about: I'm training the neural network on sample data, and I've got 1,000 samples. The training consists of gradually changing the weights and biases to make the cost function result in a smaller cost.

My question: Should I be changing the weights/biases on every single sample before moving on to the next sample, or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

  • $\begingroup$ I think you should do a little more research yourself, this is one of the most beginner question. The better question would have been what would be the difference in the 2 methods in terms of Learning? $\endgroup$ – DuttaA May 25 at 5:51
  • 1
    $\begingroup$ @DuttaA This is a perfectly reasonable question to ask. We were all beginners once (and still are in almost everything). $\endgroup$ – DrMcCleod May 25 at 7:01
  • $\begingroup$ @DrMcCleod nope.IMO This information is literally lying around, only a simple Google search is needed...it is tough to visualize differentiation or how things are working as a beginner, but a question which asks whether we do this or we do that and the information is just a Google search away reflects lack of research which is not SE guidelines. $\endgroup$ – DuttaA May 25 at 7:23
  • 3
    $\begingroup$ Possible duplicate of Is back propagation applied for each data point or for a batch of data points? $\endgroup$ – nbro May 25 at 13:05
  • $\begingroup$ The question you've asked is answered in the same video series you've linked. $\endgroup$ – Nyos May 25 at 15:17

Should I be changing the weights/biases on every single sample before moving on to the next sample,

You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the dataset before working through it each time.

or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start applying them to the network?

You can do this, it is called batch gradient descent, or in some cases (especially in older resources) just assumed as the normal approach and called gradient descent.

Each approach offers advantages and disadvantages. In general:

  • SGD makes each update sooner in terms of amount of data that has been processed. So you may need less epochs before converging on reasonable values.

  • SGD does more processing per sample (because it updates more frequently), so is also slower in the sense that it will take longer to process each sample.

  • SGD can take less advantage of parallelisation, as the update steps mean you have to run each data item serially (as the weights have changed and error/gradient results are calculated for a specific set of weights).

  • SGD individual steps make typically only very rough guesses at the correct gradients to change weights in. This is both a disadvantage (performance of the NN against the objective on the training set can decrease as well as increase) and an advantage (there is less likelihood of getting stuck in a local stationary point due the "jitter" these random differences cause).

What happens in practice is that most software allows you to compromise between batch processing and single-sample processing, to try and get the best performance and update characteristics. This is called mini-batch processing, which involves:

  • Shuffling the dataset at the start of each epoch.

  • Working through the shuffled data, N items per time where N might vary from maybe 10 to 1000, depending on the problem and any constraints on the hardware. A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.

  • Calculate the update required for each small batch, then apply it.

This is nowadays the most common update method that most neural network libraries assume, and they almost universally will accept a batch size parameter in the training API. Most of the libraries will still call simple optimisers that do that SGD; technically it is true, the gradients calculated are still somewhat randomised due to not using the full batch, but you may find this called mini-batch gradient descent in some older papers.

  • $\begingroup$ 'A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel.' How do you determine this? I haven't seen any resource where it can be commented when the batch size is just enough for peak parallelization $\endgroup$ – DuttaA May 25 at 6:11
  • 1
    $\begingroup$ @DuttaA e.g. stackoverflow.com/questions/46654424/… and pugetsystems.com/labs/hpc/… $\endgroup$ – Neil Slater May 25 at 6:40
  • $\begingroup$ A similar question was asked before on this website. I even think it is a duplicate (even though from another slightly perspective). See: ai.stackexchange.com/q/11667/2444. $\endgroup$ – nbro May 25 at 13:07
  • $\begingroup$ "A common decision is to process the largest batch size that the GPU acceleration allows to run in parallel." -- You have it backwards! The first heuristic is to process the smallest mini-batch size that results in acceptable performance. With many models, however, you hit memory limits before you saturate hardware efficiency, so you end up running the largest model that will fit into RAM. Generally, though, smaller batch sizes find better minima since they exhibit more stochasticity. A caveat is that batch-norm breaks with very small batch sizes. $\endgroup$ – Aleksandr Dubinsky May 25 at 16:27
  • $\begingroup$ @AleksandrDubinsky RAM is hardware. $\endgroup$ – DuttaA May 26 at 5:03

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical and very slow.

This is where, two optimization techniques became prominent.

  1. Mini-Batch Gradient Descent
  2. Stochastic Gradient Descent (SGD)

In mini-batch gradient descent, you use a batch size that is considerably less than total no. of training examples and update your weights after passing through these examples.

In stochastic gradient descent, you update the weights after passing through each training example.

Coming to advantages and disadvantages of the three methods we discussed.

  • Batch gradient descent gradually converges to the global minimum but it is slow and requires huge computing power.

  • Stochastic gradient descent converges fast but not to the global minimum, it converges somewhere near to the global minimum and hovers around that point, but doesn't converge ever to the global minimum. But, the converged point in Stochastic gradient descent is good enough for all practical purposes.

  • Mini-Batch gradient is a trade-off the above two methods. But, if you have a vectorized implementation of the weights updation and you
    are training with a multi-core setup or submitting the training to
    multiple machines, this is the best method both in terms of time for training and convergence to global minimum.

You can plot the cost function, w.r.t the no. of iterations to understand the difference between convergence in all the 3 types of gradient descent.

  • Batch gradient descent plot falls smoothly and slowly and gets stabilized and gets to global minimum.

  • Stochastic gradient descent plot will have oscillations, will fall fast but hovers around global minimum.

These are some blogs where there is detailed explanation of advantages, disadvantages of each method and also graphs of how cost function changes for all the three methods with iterations.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.