Tag Info

3

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error). On the other hand, in the Stochastic GD, we take one training sample to go one step towards the optimum, then repeat the latter for every ...

1

Summary: the loss needs to be differentiable, with some caveats. I will introduce some notation, which I hope is clear: if not I am happy to clarify. Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_\mathrm{min}$ which ...

Top 50 recent answers are included