David Blei: Feb 27th

Stochastic Variational Inference

Abstract:  We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can handle the full data, and outperforms traditional variational inference on a subset. (Further, we show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to very large data sets.

You can read the paper here.

Carl Smith and Ari Pakman: Feb. 20

We review and present new results on spike-and-slab priors to impose sparsity in regression problems.

Outline:

- Why spike-and-slab?

- Variational Bayes approximation to the posterior

- Computing hyperparameters using Empirical Bayes.

- Singular and non-singular Markov Chains for MCMC.

- Gibbs sampler for the posterior sparsity variables.

- Extension to regression with positive coefficients.

- Example application: finding synaptic weights in a dendritic tree

Garud Iyengar: Feb 13

Title: Fast first-order augmented Lagrangian algorithms for sparse optimization problems

Abstract:

In this talk we will survey recent work on fast first-order algorithms for solving optimization problems with non-trivial conic constraints. These algorithms are augmented Lagrangian algorithms; however, unlike traditional augmented Lagrangian algorithms we update the penalty multiplier during the course of the algorithm. The algorithm iterates are epsilon-feasible and epsilon-optimal in O(log(1/epsilon))-multiplier update steps with an overal complexity of O(1/epsilon). We will discuss the key steps in the algorithm development and show numerical results for basis pursuit, principal component pursuit and stable principal component pursuit.

Joint work with N. Serhat Aybat (Penn State)

Michael Sobel: Dec. 19

The statistical literature on causal inference is based on notation that expresses the idea that a causal relationship sustains a counterfactual conditional (e.g, to say that taking the pill caused John to get better means John took the pill and got better and that had he not taken the pill, he would not have gotten better). Using this notation, causal estimands are defined and methods used to estimate these are evaluated for bias.

This talk is to introduce you to this notation and literature and to point to some issues such as mediation and interference that have been addressed (at least somewhat) in the literature that may be of interest and relevance to neuroscience.

Eftychios Pnevmatikakis: Dec 5

Tomorrow at 1PM I'm going to present some overview of the recent work on approximate message passing algorithms (AMP) with applications to compressed sensing (CS). 

I'm going to start with a brief overview of message passing algorithms [1] and then show how it was used in [2] to derive an AMP algorithm for the standard CS setup (basis pursuit, lasso). 

The time permitting I'm going to briefly present some extensions of this methodology to the case of more general graphical models [3]. 

Material will be drawn from the following sources:

[1] Kschischang, Frank R., Brendan J. Frey, and H-A. Loeliger. "Factor graphs and the sum-product algorithm." Information Theory, IEEE Transactions on 47.2 (2001): 498-519.

[2] Donoho, David L., Arian Maleki, and Andrea Montanari. "Message-passing algorithms for compressed sensing." Proceedings of the National Academy of Sciences 106.45 (2009): 18914-18919.

[3] Rangan, Sundeep, et al. "Hybrid approximate message passing with applications to structured sparsity." arXiv preprint arXiv:1111.2581 (2011).

Josh Merel: Nov 28th

Tensor decompositions for learning latent variable models

Anima Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, Matus Telgarsky

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

Emanuel Ben-David: Nov 7th

High dimensional Bayesian inference for Gaussian directed acyclic graph models

Recent methodological work by Letac & Massam (2007) and others have introduced classes

of flexible multi-parameter Wishart distributions for high dimensional Bayesian inference

for undirected graphical models. A parallel analysis that universally extends these results

to the class of DAGs or Bayesian networks, arguably one of the most widely used classes of

graphical models, is however not available. The parameter space of interest for Gaussian

undirected graphical models is the space of sparse inverse covariance matrices with fixed zeros corresponding to the missing entries of an undirected graph, whereas for Gaussian DAG models it is the space of sparse lower triangular matrices corresponding to the Cholesky parameterization of inverse covariance matrices. Working with the latter space, though very useful, does not allow a comprehensive treatment of undirected and directed graphical models simultaneously. Moreover, this traditional approach does not lead to well-defined posterior covariance and inverse covariance Bayes estimates which respect the conditional independences encoded by a DAG, since these quantities lie on a curved manifold. In this

paper we first extend the traditional priors that have been proposed in the literature for Gaussian DAGs to allow multiple shape parameters. We then use a novel approach and proceed to define new spaces that retain only the functionally independent elements of covariance and inverse covariance matrices corresponding to DAG models. These spaces can be considered as projections of the parameter space of interest of DAG models on lower dimensions. We demonstrate that this parameter reduction bears several dividends for high dimensional Bayesian posterior analysis. By introducing new families of DAG Wishart and inverse DAG Wishart distributions on these projected spaces we succeed in

a) deriving closed form analytic expressions for posterior quantities that would normally

only be available through intractable numerical simulations,

 b) simultaneously providing a unifying treatment of undirected and directed Gaussian graphical model priors and comparisons thereof,

 c) posterior covariance and inverse covariance Bayes estimates which actually correspond to DAG models.