When training a deep network to learn object classification from a set like ImageNet, we minimize the cross entropy between the ground truth and the predicted categories. This is done in a supervised way. It is my understanding that you can separate categories in an unsupervised way using principal component analysis, but I have never seen that in a deep network. I am curious if this can be done easily in the last case. One possible way to do this would be to minimize a loss that favors categorization into one-hot vectors (this would only guarantee that an image is classified into a single category, rather that the correct category, though). Has this been done, or is there any reason why not?
Are there methods that allow deep networks to learn object categorization in a self-supervised way?
I did an experiment, took a trained densenet121 and kept the bottom layers. I trained the FC layer to a softmax and then to a lambda layer that normalized the vector. I trained the network with imagenet to make the outputs the most far a away from (1,1,1,1,1...1) as possible, so I would get one hot vectors. I did, but the network trained to a single category (put all in the same hot vector). Then I added a penalty that encourages to make vectors different is their features are also different. it improved a little, a dozen categories instead of one, bout noting close to a thousand, the available number.
I am posting this so nobody wastes his time in a silly idea like this one.