I was just reading through some convex optimization textbooks to hopefully improve my deep learning understanding and come up with new ideas. Halfway through, I decided to Google a bit! It's obvious that deep learning deal with nonconvex functions. Here's the question though: If deep learning is non-convex then why we apply a convex loss function such as cross-entropy or least square to solve a problem under a convex constraint? What am I missing?

  • 1
    $\begingroup$ Welcome to ai.se.....As far as I understand your question, it makes no sense...since loss function and the function to be approximated are quite independent aspects $\endgroup$ – user9947 May 23 '18 at 5:05
  • 2
    $\begingroup$ @DuttaA since the OP asks "what am I missing", that answer might be the one to propose formally, with details on the basics for future surfers with the same source of confusion... $\endgroup$ – DukeZhou May 23 '18 at 20:01
  • 1
    $\begingroup$ @DukeZhou I am not sure...But to the uneducated eyes those two looks independent..But there maybe some complex mathematical proofs $\endgroup$ – user9947 May 23 '18 at 20:03
  • $\begingroup$ ok, in other words means the approach can lead to local minimum. It can, but we just do not know other method, also it is seldom to trap in local minimum in very many dimentions of movement. $\endgroup$ – user8426627 Jun 18 '19 at 16:20

Well, you are definitely mixing two different things. Here are those bits:

  • The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not think about its differentiability or optimization aspects. We don't care what type of function it is; we just want the best fit of input data (ofcourse overfitting is not desired though). So, the space in which this fucntion lies has dimensions equal to the dimensions of input data (or number of input features). For every function fit we get some loss which basically is the distance from actual data point from the one predicted by the function fitted.

  • Loss function is the one which is defined in terms weights and biases. You can think the space in which loss function lies has dimensions equal to the number of PARAMETERS to tune (weights and biases). So, this space is very different from the one described in the first point. Now, here we do care about the differentiability of the function because we want a point where the function outputs least value (minimize the loss) given some particular values of inputs (weights and biases in this case) and by differentiating we can traverse to that "optimal" point in the function.

Hope this clears your doubt.

  • $\begingroup$ I think only really the second bullet is correct--if i generate a bunch of (x,y) data where y is some polynomial of x plus noise, then the function to be fit is really still in the set of polynomials--it doesn't really matter how many (x,y) points I make. Of course, regularization matters (how high of polynomial order should we permit?) and tanh or relu might not be the most efficient basis to approximate a polynomial, but, in the end, the only "dimension" that really matters is the number of knobs on the fitted function (weights and biases, or polynomial coefficients, or Fourier modes). $\endgroup$ – tsbertalan May 27 '18 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.