3
$\begingroup$

I'm searching for a loss function that fits my project. Actually, I have two questions, but they are in the same direction. I take a look at the definition of the root mean squared error (RMSE) and the Euclidean distance and they look the same to me. That's why I want to know the difference between the two. What would be the difference if I use RMSE as a loss function or the euclidean distance?

The second question is how to search for a loss function. I mean I know it depends on the problem and common things are MSE for regression and cross-entropy for classification, but let's say I have a specific problem, how do I search for a loss function?

I also saw that some people use a custom loss function and most of the deep learning frameworks allow us to define a custom loss function, but why would I want to use a custom one? How do I get the intuition that I need a custom loss function?

To be more specific, I'm doing a project where I need to reduce the GPS error of a vehicle. I have some vehicle data and my neural network will try to predict the longitude and latitude, so it's a regression problem. That's why I thought that the Euclidean distance would make sense as a loss function, right? Now, somehow MSE also makes sense to me because it is getting the difference between prediction and ground truth. Does this make sense to you as a professional ML engineer or data scientist? And if there would be a custom loss function that you can use, what would you suggest and why?

$\endgroup$
  • 1
    $\begingroup$ Hi Basilisk! Please, ask just one question per post! Your questions are really distinct. I am voting to close this post as too broad. $\endgroup$ – nbro Nov 15 at 15:01
  • $\begingroup$ Yeah, your original question (RMSE vs. Euclidean distance) has a clearcut answer, and the others have some kind of answer, but they need to be split out in the separate questions. $\endgroup$ – John Doucette Nov 19 at 1:58
3
$\begingroup$

For the first question, RMSE and Euclidean distance have no difference, not that i know of.

For the second question, you only need the common loss function for normal tasks.

MSE is a common loss function used in linear regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For logistic regression, use Binary Cross Entropy loss. See this for details:

Cross-entropy loss, or log loss, measure the performance of a classification model whose output is a probability value between 0 and 1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from the statement of your problems itself. In classification you work with a very particular set of possible output values thus MSE is badly defined.

To better understand the phenomena it is good to follow and understand the relations between

Cross-entropy

Logistic regression (binary cross-entropy)

Linear regression (MSE)

You will notice that both can be seen as a maximum likelihood estimator (MLE), simply with different assumptions about the dependent variable.

When you derive the cost function from the aspect of probability and distribution, you can observe that MSE happens when you assume the error follows Normal Distribution and cross-entropy when you assume binomial distribution. It means that implicitly when you use MSE, you are doing regression (estimation) and when you use CE, you are doing classification.

Source: https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up

For a custom loss function, there is examples like the triplet loss and cases where you need to optimize two loss at once, then you need a custom loss function. For the case of triplet loss, it is used in one-shot learning tasks like face recognition. There, face images are fed through a CNN to get embeddings about description of the face. In the training phase there is three images per batch. One anchor, one positive (which is the same person as anchor ) and one negative (different person from anchor). The loss function, triplet loss, maximizes the distance between the anchor embedding and the negative embedding. In reverse, it also minimize the distance between the anchor embedding and the positive embedding. The loss function is like this: enter image description here

As you can see, there is two components here, and they are subtracted to maximize the second component and minimize the first. The alpha value is to make the loss positive as the gradient descent optimizes the loss to be as near to zero as possible.

The case of style transfer also introduces a custom loss function. Here is the loss function: enter image description here

Style transfer basically alters the input image to be styled like the style image of the system.

The loss function consist of style loss and content loss. The content loss decreases the distance of the output image and the style loss decreases the distance of the output image and the style in an artistic sense. The second loss makes the output image imitate the style of the the style image.

In both scenario a custom loss function is used. In both cases, the custom loss function is used to optimize two losses together, either positively or negatively. A weight can also be introduced to weight the two losses as shown in the second case. However the second case introduce a custom loss function. That leads to the second use of custom loss function. They can be used for specific tasks and optimize a specific goal. The first use for custom loss function can be explored but the second requires careful engineering and research.

So in short, custom loss function are used for either combining two loss or a specific task.

Hope I can help you.

$\endgroup$
  • $\begingroup$ thnx for your answer but with all respect I don't know how does that answer my questions. I mean I'm not a beginner with NNs,I already know the differences between mse/rmse and log loss and why is mse used for regression and cross entropy for classification. I also wrote in the question that I knew that. My question is about difference between rmse and euclidean direction and why would someone need to use custom loss if the standard losses works fine and are differentiable, why or when should I use a custom loss or how I get the intuition of using custom loss. that was my question $\endgroup$ – basilisk Nov 15 at 10:31
  • $\begingroup$ Sorry didn't add the final part. Wait a second ediiting my question. Thank you for your reminder @basilisk $\endgroup$ – Clement Hui Nov 15 at 10:33
  • $\begingroup$ I have changed the answer. Sorry for didn't adding it at first. Hope it can help you @basilisk $\endgroup$ – Clement Hui Nov 15 at 11:37
  • 1
    $\begingroup$ So if a custom loss is used it is used for optimizing two different goals , not only optimize two things. The two goals maybe one positive one negative, or weighted loss. In your case of two output, you can just use a conventional loss function and optimize the output using the target as the two output have the same "goal" like the objective is to minimize the distance between the output and label, not one minimize one maximize. You don't need a custom loss function for this. $\endgroup$ – Clement Hui Nov 15 at 12:27
  • 1
    $\begingroup$ @basilisk I just wanted to add an example of a custom loss function. Ordinal Categorical Classification; meaning when order of the classes matter: given classes 1 to 5, identifying 1 as 2 is not as bad as identifying in as 5. There is no loss function for such a problem in Keras, so one has to use a custom function, or use regular classifier which do not use all the information available. $\endgroup$ – serali Nov 15 at 12:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.