If I was Given a set of large training examples (xi,yi), how can training a neural network (NN) via stochastic gradient descent differs from using regular gradient descent in terms of the mathematical expressions used to represent the loss in each approach. You may assume use of sum square loss and that the predictions of the neural network are hΘ(xi) for each input example xi and current weights (parameters) Θ

  • $\begingroup$ How are you initializing your network? That, at least in my experience, is what the "Stochastic" in SGD refers to. $\endgroup$ Jan 8 at 14:08
  • $\begingroup$ Just wanted to know how it differs from regular gradient decent $\endgroup$
    – Allen R
    Jan 8 at 14:16
  • 2
    $\begingroup$ Your question is not an exact duplicate of the linked question, but the answer there contains the answer to your question. In general, before asking a question, you should do a little bit of research. The answer to this question can be found easily on the web and it's been asked many times. Additionally, next time, please, put your specific question in the title. "Deep learning and machine learning" is like the most generic title that you could have chosen, which gives zero information about your actual question. $\endgroup$
    – nbro
    Jan 8 at 14:27
  • $\begingroup$ While I completely agree with your comments @nbro, I'm not sure it's exactly the same... That question is more accurately, "What's the difference between SGD and Minibatch SGD." $\endgroup$ Jan 8 at 20:40
  • 1
    $\begingroup$ @DavidHoelzer You're right. People differentiate between mini-batch and "regular" GD. I've added the link to another question that asks about the difference between mini-batch and batch GD. Duplication was not the only reason why I closed this post. It also seems that this is also homework and the OP didn't put any effort into trying to answer his question. We could have a canonical question that asks "what is the difference between GD, SGD, mini-batch and batch GD", but this question was too poor to become that canonical question. If you want to ask that question, feel free to do it. $\endgroup$
    – nbro
    Jan 8 at 21:06