If I want to find a (linear) subspace onto which a data-set projects well, I can simply use PCA. However, often the data can project with much smaller error if I first separate it into a couple of classes and then perform PCA for each class individually. But what If I don't know what kind of classes there might be in my data and into how many classes it would make sense to split the data? What kind of machine learning algorithm can do this well?

Example: enter image description here

If I'd just cluster first based on distance in the high-dimensional space, I would arrive at the bad clustering. There are 5 clusters and the green and red clusters don't project very well onto a 2D subspace.

As a human looking at the data, I see however that if I separate the data as indicated, red and blue will project very well onto a plane each and green will project very well onto a line, so I can run PCA for each group individually.

How can I automate this clustering based on how well it will project onto as low-imensional subspaces as possible?

Something like minimize E = SumOverClusters(SumOverPoints(SquaredDist(projected_point, original_point)) * (number_dims_projected / number_dims_original)) + C * number_of_clusters

What technique is well suited to do that?

(edit: while the example shows a 3d space, I'm more interested in doing that in about 64dimensional spaces)

  • $\begingroup$ Searching for "PCA mixture model" gives quite a few hits, is that maybe what you are looking for? (Effectively just combining mixture of experts + PCA) $\endgroup$ – Hyperplane Jul 31 '20 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.