Nonparametric Bayesian Discrete Latent Variable Models For Unsupervised Learning

Author : Ke Jiang
Publisher :
ISBN 13 :
Total Pages : 123 pages
Book Rating : 4.:/5 (99 download)

DOWNLOAD NOW!

Download or read book Small-variance Asymptotics for Bayesian Models written by Ke Jiang and published by . This book was released on 2017 with total page 123 pages. Available in PDF, EPUB and Kindle. Book excerpt: Bayesian models have been used extensively in various machine learning tasks, often resulting in improved prediction performance through the utilization of (layers of) latent variables when modeling the generative process of the observed data. Extending the parameter space from finite to infinite-dimensional, Bayesian nonparametric models can infer the model complexity directly from the data and thus also adapt with the amount of the observed data. This is especially appealing in the age of big data. However, such benefits come at a price: the parameter training and the prediction are notoriously difficult even for parametric models. Sampling and variational inference techniques are two standard methods for inference in Bayesian models, but for many problems, neither approach scales effectively to large-scale data. Currently, there is significant ongoing research trying to scale these methods using ideas from stochastic differential equations and stochastic optimization. A recent thread of research has considered small-variance asymptotics of latent-variable models as a way to capture the benefits of rich probabilistic models while also providing a framework for designing more scalable combinatorial optimization algorithms. Such models are often motivated by the well-known connection between mixtures of Gaussians and K-means: as the variances of the Gaussians tend to zero, the mixture of Gaussians model approaches K-means, both in terms of objectives and algorithms. In this dissertation, we will study small-variance asymptotics of Bayesian models, yielding new formulations and algorithms which may provide more efficient solutions to various unsupervised learning problems. Firstly, we consider clustering problems: exploring small-variance asymptotics for exponential family Dirichlet process (DP) and hierarchical Dirichlet process (HDP) mixture models. Utilizing connections between exponential family distributions and Bregman divergences, we derive novel clustering algorithms from the asymptotic limit of the DP and HDP mixtures that features the scalability of existing hard clustering methods as well as the flexibility of Bayesian nonparametric models. Secondly, we consider sequential models: exploring the small-variance asymptotic analysis of the infinite hidden Markov models, yielding a combinatorial objective function for discrete-data sequence observations with a non-fixed number of states. This involves a k-means-like term along with penalties based on state transitions and the number of states. We also present a simple, scalable, and flexible algorithm to optimize it. Lastly, we consider the topic modeling problems, which have emerged as fundamental tools in unsupervised machine learning. We approach it via combinatorial optimization, and take a small-variance limit of the latent Dirichlet allocation model to derive a new objective function. We minimize this objective by using ideas from combinatorial optimization, obtaining a new, fast, and high-quality topic modeling algorithm. In particular, we show that our results are not only significantly better than traditional small-variance asymptotic based algorithms, but also truly competitive with popular probabilistic approaches.