What is the effectiveness of pre-training of unsupervised deep learning?
Does unsupervised deep learning actually work?
Unsupervised pre-training was done only very shortly, as far as I know, at the time when deep learning started to actually work. It extracts certain regularities in the data, which a later supervised learning can latch onto, so it is not surprising that it might work. On the other hand, unsupervised learning doesn't give particularly impressive results in very deep nets, so it is also not surprising that with current very deep nets, it isn't used anymore.
I was wondering whether the initial success with unsupervised pre-training had something to do with the fact that the ideal initialization of neural nets was only worked out later. In that case, unsupervised pre-training would only be a very complicated way of getting the weights to the correct size.
Unsupervised deep learning is something like the holy grail of AI right now and hasn't been found yet. Unsupervised deep learning would allow you to use massive amounts of unlabeled data and let the net form its own categories. Later you can just use a little bit of labeled data to give these categories their proper labels. Or just train it immediately on some task, in the conviction that it has a huge amount of knowledge about the world already. This is also what the problem of common sense comes down to: a huge and detailed model of the world, that could only be acquired by unsupervised learning.