I have two convex, smooth loss functions to minimise. During the training (a very simple model) using batch SGD (with tuned optimal learning rate for each loss function), I observe that the (log) loss curve of the loss 2 converges much faster and is much more smooth than that of the loss 2, as shown in the figure.

What can I say more about the properties of the two loss functions, for example in terms of smoothness, convexity, etc?

enter image description here

  • $\begingroup$ I'll ask the same question I asked in the data science SE : what make you think you can get overall properties of functions with a set of evaluations along an optimisation path ? From just your graph it's not possible to tell that much, yeah the minima of loss 1 appears lower than the one of loss 2. The path appears bumpier but it may be due on how you selected your 'optimal' learning rates. Even with a lot more details we probably won't be able to answer that question. $\endgroup$ – lcrmorin Jan 5 at 0:59
  • $\begingroup$ The loss 1 is in fact smaller than the loss 2, by construction. As I said, the learning rate seems to impact only the speed of convergence, not the smoothness of the loss curve. I've also tried other datasets and observed the same behaviors. I know that it's not enough to generalise from what I observe, but I want to understand why such behaviors persist in the datasets I'm working with. $\endgroup$ – SiXUlm Jan 5 at 1:16
  • $\begingroup$ Maybe start by giving us the form of your loss and be more precise about what you want to know (that would be not trivially given by the loss forms). $\endgroup$ – lcrmorin Jan 5 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.