Josh Merel: Nov 28th

Tensor decompositions for learning latent variable models

Anima Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, Matus Telgarsky

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

Roy Fox: July 31st

"Residual Component Analysis: Generalising PCA for more flexible inference in linear-Gaussian models"

http://icml.cc/2012/papers/114.pdf

Abstract:

Probabilistic principal component analysis (PPCA) seeks a low dimensional representation of a data set in the presence of independent spherical Gaussian noise, Σ = σ^2I. The maximum likelihood solution for the model is an eigenvalue problem on the sample covariance matrix. In this paper we consider the situation where the data variance is already partially explained by other factors, e.g. conditional dependencies between the covariates, or temporal correlations leaving some residual variance. We decompose the residual variance into its components through a generalised eigenvalue problem, which we call residual component analysis (RCA). We explore a range of new algorithms that arise from the framework, including one that factorises the covariance of a Gaussian density into a low-rank and a sparse-inverse component. We illustrate the ideas on the recovery of a protein-signaling network, a gene expression time-series data set and the recovery of the human skeleton from motion capture 3-D cloud data.

Johaness Bill: July 16th

Probabilistic inference and autonomous learning in recurrent networks of spiking neurons 

Numerous findings from cognitive science and neuroscience indicate that mammals learn and maintain an internal model of their environment, and that they employ this model during perception and decision making in a statistically optimal fashion. Indeed, recent experimental studies suggest that the required computational machinery for probabilistic inference and learning can be traced down to the level of individual spiking neurons in recurrent networks. 

At the Institute for Theoretical Computer Science in Graz, we examine (analytically and through computer simulations) how recurrent neural networks can represent complex joint probability distributions in their transient spike pattern, how external input can be integrated by networks to a Bayesian posterior distribution, and how local synaptic learning rules enable spiking neural networks to autonomously optimize their internal model of the observed input statistics. 

In the talk, I aim to discuss approaches of how recurrent spiking networks can sample from graphical models by means of their internal dynamics, and how spike-timing dependent plasticity rules can implement maximum likelihood learning of generative models.

Carl Smith : Dec 15

This Wednesday I'll pick up where we left off last week when we covered graphical models, exponential families, and the basic ideas behind variational inference. This week I will go over variational inference in greater depth, and then describe some approximations to the variational problem that render it tractable: sum-product and the Bethe entropy approximation; mean field methods (time permitting); and convex approximations, in particular tree-reweighted belief propagation. The material is drawn from chapters 3, 4, 5, and 7 of the same paper.

Carl Smith : Dec 1

"This week I plan to present topics from the first half (chapters 1-5) of Wainwright and Jordan's "Graphical Models, Exponential Families, and Variational Inference". I will emphasize the ideas of 1) conjugate duality between partition function and negative entropy, and 2) nonconvexity in mean field approaches to inference. I will present the following week on some combination of ideas from the second half of the same paper, related papers by Wainwright, and related stuff Liam and I have been working on, depending on time and what people are interested in after the first hour."