Beyond maximum likelihood and density estimation: A sample-based criterion for
unsupervised learning of complex models
The goal of many unsupervised learning procedures is to bring two probability
distributions into alignment. Generative models such as Gaussian mixtures and
Boltzmann machines can be cast in this light, as can recoding models such
as ICA and projection pursuit. We propose a novel sample-based error measure
for these classes of models, which applies even in situations where maximum
likelihood (ML) and probability density estimation-based formulations cannot
be applied, e.g., models that are nonlinear or have intractable posteriors.
Furthermore, our sample-based error measure avoids the difficulties of
approximating a density function. We prove that with an unconstrained model,
(1) our approach converges on the correct solution as the number of samples
goes to infinity, and (2) the expected solution of our approach in the
generative framework is the ML solution. Finally, we evaluate our approach
via simulations of linear and nonlinear models on mixture-of-Gaussians and ICA
problems. The experiments show the broad applicability and generality of our
approach.
Retrieve Paper (pdf)