I have a task where I need to take "data points" which consist of collections of items. Each item needs to be categorised according to predefined categories. That's the easy part - my solution is to train a deep neural network with cross entropy loss. By the way, the reason I don't classify each item separately is because they acquire their meaning when they come together as a set.

The hard part is that each of these items also have a cluster label. Each cluster can only have items of one category in it, and there can be any number of clusters. Unsupervised clustering methods (applied after the neural network does the categorisation) work fairly well, but not well-enough for my needs. I'd like to:

A. Make use of the fact that I have the ground truth labelling for these clusters

B. Leverage my deep neural network because a lot of the "reasoning" required to solve the classification task will be conducive to the clustering task.

Answers which address at least one of those are useful to me.


I realised I might be confusing people with this concept of cluster "labels". To clarify, this is no different than the standard way a classical unsupervised clustering algorithm might return its results. If I have N data points and feed them to a clustering algo, the algo might return N labels, one for each data point, and each of which are integers in [0, C-1] where C is the number of clusters. In my example we have the labels for a training dataset and want to make use of them during training. We cannot use softmax + cross-entropy loss because the cluster labels are permutation invariant.

  • $\begingroup$ Is there a reason you can't create two separate output layers, one that tells you the category and the other that tells you the cluster? Both outputs would use crossentropy loss. In effect, this would be a supervised clustering, if you will. $\endgroup$ Sep 2 at 23:12
  • $\begingroup$ @DavidHoelzer I had a similar question on stats. I think I somehow confused people with my wording. The cluster "labels" are permutation invariant, so cross-entropy loss won't work. $\endgroup$ Sep 3 at 8:52
  • $\begingroup$ Just to spell it out for anyone who comes across this, the reason permutation invariance is a problem is because we shouldn't punish the network for outputting labels [1, 1, 0, 0] when the ground truth is [0, 0, 1, 1]. These label sets are equivalent in terms of clustering. $\endgroup$ Sep 3 at 9:03
  • 1
    $\begingroup$ @user253751 it certainly will impact training. It will raise similar issues to training a dog vs canis_lupis binary classification task on a dataset of dog images. There are no distinguising traits, so the network will never learn your labels. It will just learn to predict the distribution of dog:canis_lupis randomly (unless there's a leak, and there's some way of distinguishing). In the clustering task, say I always label the biggest cluster (whatever it is) with 0s, then the network would probably just learn to mostly predict 0s... (and maybe it will learn some vague notion of clusters) $\endgroup$ Sep 3 at 11:13
  • 1
    $\begingroup$ It's not exactly the same... but hopefully you get the point. There's just no concrete way to say which cluster should be 0 and which should be 1 and... so on for each input. So it will just "confuse" the model. $\endgroup$ Sep 3 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.