Stefano Ermon from Stanford gave a good talk at SF Big Analytics this week. One of his topics was how you can use a combination of a discriminative model with a generative model to get better results in situations when you have reasonable volume of unlabeled data but a small volume of labeled data.
The basic technique is:
- Train a generative model to generate simulated observations, for this you only need unlabeled data
- Train a discriminative model to try and classify observations as either genuine or simulated
- Have the two models compete until the generative model wins and can produce simulated observations that are virtually indistinguishable from the genuine models.
- This generative model now has knowledge about the latent variables that drive the generation of the observations
- Now you can use a combination of that knowledge about the latent variables and your small quantity of labeled observations to train a new model to be able to successfully label unlabeled observations.
These latent variables may be semantically meaningful but they don’t have to be. One way to build on this approach is to develop techniques for human supervision of the latent variable selection to ensure that they are semantically meaningful.
The benefits of such an approach potentially go beyond handling a lack of labeled data. To the degree we can produce semantically meaningful latent features then we would have all the benefits of explainability including better troubleshooting and improved ability to estimate generalizability.
More info on the Stanford team’s technique is in this PDF.
One thought on “Latent Variable Models for Semi-Supervised Learning”