1
$\begingroup$

So I want to classify 3 classes and I am using CNN.

So if I see my data visually, I can tell that there are two major features/differences between class 1 vs. (class 2 and class 3). And then for class 2 and class 3, there are one major feature can be used to differentiate.

And I am applying softmax at the end so I will have 3 probability distributions and one for each class. And I am reporting the index with the highest probability to be my class. However, I am not sure if this is a logical way of doing so, because my major features are not equally distributed between classes.

Any suggestions on how should I classify these 3 classes? Or any other ideas of confidence evaluation. Thanks in advance.

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you provide a little more clarification? What do you mean by your features are not equally distributed between classes? I assume by what you mean by "classify these 3 classes" you are feature engineering, which doesn't really apply to CNNs. $\endgroup$
    – Noah Lott
    Commented Jul 30 at 20:14
  • $\begingroup$ @NoahLott Thank you for pointing out your confusion. By saying features not equally distributed, I am referring to the number of features should be used for classification are not the same. For example, both class 2 and 3's time-series data have significant drop (one feature) compared to class 1. But the drop of class 2 and 3 compared to each other are not significant. And then class 2 and 3's data only have 1 major different characteristic/feature. $\endgroup$
    – L Z
    Commented Jul 30 at 20:46

1 Answer 1

Reset to default
0
$\begingroup$

Based on your true inductive bias, in theory it's better to use hierarchical classification.

In the field of machine learning, hierarchical classification is sometimes referred to as instance space decomposition, which splits a complete multi-class problem into a set of smaller classification problems.

A concrete example is described in Redmon et al's “YOLO9000: Better, Faster, Stronger” (2017). In your case the idea is that before training the model you propagate the labels of your class 2 and 3 to their parent, say, as label 1' which is a sibling of your label 1, then you train a multi-label classifier on such label-transformed tree-like data of multiple levels where you enforce mutual exclusion constraint at each level. In your case it will perform hierarchical classification to predict a set of labels at just two levels of abstraction. Also in this way you can also hedge your bet to reject decision if the probability threshold to further classify label 2&3 is low.

Hierarchical classification. ImageNet labels are pulled from WordNet, a language database that structures concepts and how they relate. In WordNet, “Norfolk terrier” and “Yorkshire terrier” are both hyponyms of “terrier” which is a type of “hunting dog”, which is a type of “dog”, which is a “canine”, etc. Most approaches to classification assume a flat structure to the labels however for combining datasets, structure is exactly what we need... The final result is WordTree, a hierarchical model of visual concepts. To perform classification with WordTree we predict conditional probabilities at every node for the probability of each hyponym of that synset given that synset... Using the same training parameters as before, our hierarchical Darknet-19 achieves 71.9% top-1 accuracy and 90.4% top-5 accuracy.

Finally even if you still decide to use a flat classifier, it's not a good way to simply report the index with the highest probability to be your class. Assuming no test data shifts, at least you need to implement temperature scaling or other calibration techniques to ensure the softmax probabilities are well-calibrated and reflective of true confidence based on your validation set. Temperature scaling empirically produces the lowest expected calibration error (ECE) on a variety of DNN classification problems and is much simpler and faster than other methods.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for getting back to me @cinch. I just have a follow up question. So let's say I have 2 super classes and one has only 1 sub class and the other has 2. Then for my model, I will have 2 separate softmax outputs one with size 2 (for super classes) and the other one with size 3 (for sub classes)? For the outputs, for example, I will check which super classes does it classified as, if it is super class 2, then the conditional prob. would be [0, P(sub class 1) * P(super class 2), P(sub class 2) * P(super class 2)] and pick the max prob's index as my final class? $\endgroup$
    – L Z
    Commented Aug 6 at 16:50
  • $\begingroup$ I guess I am just a bit confused about how to train the model as hierarchical classification. Like should the last layer of the fully connected layer be set as number of my super class and then followed by another fully connected layer to be number of my sub class? So I will return 2 set of probability distribution? But then during training, should I just add the two set of losses and then apply the back propagation? $\endgroup$
    – L Z
    Commented Aug 7 at 15:19
  • $\begingroup$ For specific model implementation here's one way for your simple case (you can check my linked YOLO9000 paper for more details): use a shared feature extraction layer followed by separate softmax layers for superclass and subclass predictions. Then compute separate losses for superclass and subclass predictions and combine them. Use backpropagation on the combined loss to train the model. Essentially you define your own combined loss and your own hierarchical classifier model's __init__() and forward() steps which are to be used in your epoch training loop. $\endgroup$
    – cinch
    Commented Aug 7 at 17:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .