Rajesh Ranganath: February 25th

Title: "Black Box Variational Inference"

Abstract: Variational inference has become a widely used method to approximate posteriors in complex latent variables models. However, deriving a variational inference algorithm generally requires significant model-specific analysis, and these efforts can hinder and deter us from quickly developing and exploring a variety of models for a problem at hand. We present a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation. Our method is based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the variational distribution. We develop a number of methods to reduce the variance of the gradient, always maintaining the criterion that we want to avoid difficult model-based derivations. We evaluate our method against the corresponding black box sampling based methods. We find that our method reaches better predictive likelihoods much faster than sampling methods. Finally, we demonstrate that Black Box Variational Inference lets us easily explore a wide space of models by quickly constructing and evaluating several models of longitudinal healthcare data.

Josh Merel: Feb 18th

ADADELTA and LSTMs

We will discuss the adadelta paper
http://arxiv.org/pdf/1212.5701v1.pdf

and talk about LSTM layers. The original paper is (for reference): http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
But we will probably discuss a more recent result using LSTM layers that uses a slightly different notation.

Uygar Sümbül: Feb 4th

Towards automated segmentation of neurons from Brainbow images

A necessary step to learn how neural circuits account for
observed behaviors in health and disease is to map the connectivity of
individual neurons. Stochastic expression of fluorescent proteins with
different colors where individual cells express one of many
distinguishable hues – the Brainbow method – has generated striking
images of nervous tissues. However, its use has been limited. One
basic shortcoming has been the various noise sources in fluorescence
expression within individual neurons. Here, we propose a method to
automate the segmentation of neurons in Brainbow image stacks using
spectral clustering.

 

Brian DePasqual: Jan 28

Embedding low-dimensional continuous dynamical systems in recurrently connected spiking neural networks

Despite recent advances in training recurrently connected firing-rate networks, the application of supervised learning algorithms to biologically plausible recurrently connected spiking neural networks remains a challenge. Such models, when trained to directly replicate neural data, hold great promise as powerful tools for understanding dynamic computation in biologically realistic neural circuits. In this talk I will discuss our progress in the training of recurrently connected spiking networks, the application of our training framework to neural population data and a novel interpretation of continuous neural signals that arises within the context of these models.

     Extending the iterative supervised learning algorithm of Sussillo & Abbott [2009], we have made several critical observations about the conditions necessary for successfully training recurrent spiking networks. Due to their impoverished short-term memory, multiple signals that form a “dynamically complete” basis must be trained simultaneously for successful training. I will illustrate this by showing a variety of examples of spiking neural networks replicating the dynamics of both autonomous and non-autonomous linear and non-linear continuous dynamical systems. Additionally, I will discuss recent efforts to incorporate a variety of network optimization constraints such that the learned connectivity matrices obey common constraints of biological networks, including sparsity and Dale’s Law. Finally, I will discuss our efforts to fit spiking models to population data from the isolated nervous system of the leech.

     Once trained, our models can be viewed as a low-dimensional, continuous dynamical system - traditionally modeled with firing-rate networks - embedded in a high-dimensional, spiking dynamical system. In light of this view, I will present a novel interpretation of firing-rate models and smoothly varying signals in general. Traditionally a continuous neural signal modeled as a “firing-rate unit” was a simplified representation of a pool of identical but noisy spiking neurons. In our formulation, each continuous neural signal represents an overlapping population of spiking neurons and is thus more akin to the multiple, continuous population trajectories one would uncover from experimental data via dimensionality reduction. By allowing these continuous signals to be constructed from overlapping pools of spiking neurons, our framework requires far fewer spiking neurons to arrive at the equivalent, traditional rate description.

Daniel Soudry: Jan 21st

Daniel Soudry will talk about the following paper:

Title: Fixed-form variational posterior approximation through stochastic linear regression

Authors:Tim Salimans and David A. Knowles

Abstract: We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method can be used to approximate any posterior distribution, provided that it is given in closed form up to the proportionality constant. The approximation can be any distribution in the exponential family or any mixture of such distributions, which means that it can be made arbitrarily precise. Several examples illustrate the speed and accuracy of our approximation method in practice.

Moritz Deger: Dec 17th

Dynamics and estimation of large-scale spiking neuronal network models

Computations in neural circuits emerge from the interaction of many neurons. Although accurate single neuron models exist, little is known about the synaptic connectivity of large-scale circuits of neurons. However, recent experimental techniques enable recordings of the activity of thousands of neurons in parallel. Such data hold the promise that inferences about the underlying synaptic connectivity can be made. I will present two complementary approaches to gain insights into the collective dynamics of neural circuits.
On the one hand, I will report on recent progress in the reconstruction of networks of 1000 simulated spiking neurons by maximum likelihood estimation of a generalized linear model (GLM), in which a million possible synaptic efficacies have to be determined. The work demonstrates that reconstructing the connectivity of thousands of neurons is feasible, and that hidden embedded subpopulations can be detected in the reconstructed connectivity.

On the other hand, spiking GLM's are a versatile class of single neuron models that can represent important features such as intrinsic stochasticity, refractoriness, and spike-frequency adaptation. For this class of models, I will present a new theory of population dynamics, which takes into account finite-size fluctuations and accurately describes the population-averaged neural activity in time, in response to arbitrary stimuli. Based on this theory, GLM network models with parameters extracted from data can be used to understand neural processing in realistic networks, and circuit-level information processing can be explained.

Post-NIPS special discussion: Dec 16th 2-3:30pm