0
$\begingroup$

I am considering a rather typical regression problem, but, for practice, I am trying to implement this as a classification problem.

The setup is as follows. I have $\mathbb{R}$-valued labels $y_i \in [-1,1]$, which I then discretize to $N$ buckets -- my classification problem is to then predict the labels to the nearest bucket.

This is rather straightforward and easy to implement with a cross-entropy loss function. However, I do not believe that this is the best option, as I would ideally like my predictions to be close to their correct bucket, even if I do not predict them correctly (which will be more difficult as if I take $N$ larger).

My current approach involves using a mean-squared error loss function. My network outputs logits for each bucket, I apply a softargmax (so the network remains differentiable) and then convert the output of the network into the $\mathbb{R}$-valued prediction.

My (very premature) results are nothing to write home about. So, I ask, is there a more natural loss function that I could consider for this exercise?

$\endgroup$
1
  • 1
    $\begingroup$ What code are you using ? share what you have tried $\endgroup$ – FabioSpaghetti May 27 at 9:47
0
$\begingroup$

Bucketing is usually used in industry because we are not sure what we are going to get. Some amount of observation on the distribution will give you a good idea of bucketing. But, there is caveat, if there isn't a lot of variance then bucketing won't help much.

Also, in terms of losses, since different buckets will have different number of examples, class imbalance will arise. So, focal loss is the better loss here.

I hope you get results you can write home about.

$\endgroup$
2
  • $\begingroup$ Hi, thanks for your input. I have thought of another approach, and would like to perhaps get your opinion. I am still running a classification based on my buckets, but I am using a KL divergence loss function. Additionally, for the labels I am "adding" noise to them -- essentially, centring a truncated normal (so that the pdf remains in my domain) around the observed label. In this sense, I am training a network to be a "soft" classifier. Does this make sense to you, and do you know if there is any existing approaches like this? The results are rather good for me. $\endgroup$ – reimama Jun 2 at 14:48
  • $\begingroup$ The approach seems a bit convoluted. You are first assuming a discrete distribution over continuous values then you are using losses suitable for distributions (it is working on the label distribution). Also, the noise part is easily there in cross entropy function (see label smoothing). This can be a experiment and may even perform. But, traditionally, the pipeline would be (continuous -> discrete -> NN with cross entropy (with label smoothing for regularization). Do tell how that worked out for you. $\endgroup$ – Abhishek Verma Jun 3 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.