Is there a bias-variance equivalent in unsupervised learning?

Question

In supervised learning, bias, variance are pretty easy to calculate with labeled data. I was wondering if there's something equivalent in unsupervised learning, or like a way to estimate such things?

If not, how do we calculate loss functions in unsupervised learning?

Answer 1

Yes, the concept applies but it is not really formalized. Consider unsupervised learning as a form of density estimation or a type of statistical estimate of the density.

Variance: You will train on a finite sample of data selected from this probability distribution and get a model, but if you select a different random sample from this distribution you will get a slightly different unsupervised model. This variation caused by the selection process of a particular data sample is the variance.

Bias: This is a little more fuzzy depending on the error metric used in the supervised learning. An unsupervised learning algorithm has parameters that control the flexibility of the model to 'fit' the data. For example, k means clustering you control the number of clusters. Simple example is k means clustering with k=1. You could imagine a distribution where there are two 'clumps' of data far apart. The mean would land in the middle where there is no data. This model is biased to assuming a certain distribution. For a higher k value, you can imagine other distributions with k+1 clumps that cause the cluster centers to fall in low density areas. For a low value of parameters, you would also expect to get the same model, even for very different density distributions. This is also a form of bias. This unsupervised model is biased to better 'fit' certain distributions and also can not distinguish between certain distributions.

You can see that because unsupervised models usually don't have a goal directly specified by an error metric, the concept is not as formalized and more conceptual.