When you predict Start Index, end Index in Question Answering NLP task (SQUAD Data), you use CrossEntropy as a loss function. Won't it be same if you used MSE or RMSE for the same because at the end of the model, all we're trying to find is an absolute number from the given max_length (suppose 300). We could Regress this number?

2 problems I can think of are:

  1. MSE or RMSE will give unbounded values in the starting (for example 4321) even though we have a max length of 300 but I think it'll settle down eventually with training.
  2. If the original Index was 5 logit could predict 4 which is Very okay for MSE as it's just 1.0 points away (worse could 300) but according to CrossEntropy, it's as bad as predicting 300.

I just want to know what other problems can be there if we used and when one should be used in place of other?

For example in Binary Image Segmentation, both can be used but BinaryCrossEntropy would give better results as the boundary is strict 0-1 etc.

Apart from the very obvious, If you look at

  1. MSE is Cross Entropy at heart
  2. Comparing Sigmoid-MSE With Softmax Cross-Entropy for Image Classification
  3. Can we use cross entropy to solve the regression problem?

You'll get a sense that you can use them but there is a caveat, for example in the first link, it is beautifully explained that:

For example, there is an application of MSE loss in a task named Super Resolution in which (as the name suggests) we try to increase the resolution of a small image as best as possible to get a visually appealing image. If we use MSE loss alone, the final image will be really blurry and not appealing. Knowing that MSE is a kind of Cross Entropy where we assume that our target distribution is Gaussian, we can easily find out the reason why our model produces these blurry images: because it assumes that pixel data is normally distributed and always picks the values which have the most probability; i.e. the values from the middle of the bell-curve. So, it rarely uses the values which make the image sharp and appealing because they are far from the middle of the bell-curve and have really low probabilities. Our model becomes conservative in a sense that when it doubts what value it should pick, it picks the most probable ones which make the image blurry!



You must log in to answer this question.