Gustavo Lacerda: March 20th

Title: spatial regularization

Consider modeling each neuron as a 2-parameter logistic model (spiking probability as a function of stimulus intensity), and suppose we perform independent experiments on each neuron. Now imagine that the data isn't very informative, so we need to regularize our estimates. We can do spatial regularization by adding a quadratic penalty on the difference of estimates for nearby neurons. Now, suppose that there are *two* types of neurons, and that you only want to shrink together neurons of the same type. We don't want our estimate to be influenced by "false neighbors", i.e. neurons that are spatially close but of a different type. We discuss how to optimize this model. Finally, we explore the idea of Fused Group Lasso.

Kolia Sadeghi : Sept. 20

This week, I'll be giving a fly-by overview of a string of recent papers on exact sparse signal recovery that do better than LASSO by solving a sequence of L1 or L2 penalized problems.  Here is a basic narrative:

LASSO uses a penalty weighted by the same lambda for all coefficients.  What happens if you assign different lambdas to each coefficient, and update these lambdas iteratively?  Candes and Boyd do this in Enhancing sparsity by reweighted L1 minimization

You can obtain sparsity by iterative reweighting even for L2-penalized problems: if some of the lambdas become infinite, the corresponding coefficients become exactly zero.  Chartrand and Yin find a particularly good L2 reweighing scheme in Iteratively reweighted algorithms for compressive sensing

All of the above methods reweigh each lambda based only on the value of its corresponding coefficient: they are separable.  In Iterative reweighted l1 and l2 methods for finding sparse solutions, Wipf considers non-separable reweighting schemes that come out of Sparse Bayesian Learning (SBL), which you might also know by the name of Relevance Vector Machine or Automatic Revelance Determination.

Alex Ramirez: June 21th

Alex will be presenting a short version of this paper. In it the authors consider loss functions, for many estimators, that obey certain smoothness and convexity requirements and prove a global, geometric convergence (fast) rate of convergence under Nestorv's Gradient descent method up to a level of Statistical precision.

There will be no meeting on June 14th.

Kamiar Rahnama Rad: June 7th

Information rates and Optimal decoding in Large Populations

Many fundamental questions in theoretical neuroscience involve optimal decoding and the computation of Shannon information rates in populations of spiking neurons. In this paper, we apply methods from the asymptotic theory of statistical inference to obtain a clearer analytical understanding of these quantities. We find that for large neural populations carrying a finite total amount of information, the full spiking population response is asymptotically as informative as a single observation from a Gaussian process whose mean and covariance can be characterized explicitly in terms of network and single neuron properties. The Gaussian form of this asymptotic sufficient statistic allows us in certain cases to perform opti- mal Bayesian decoding by simple linear transformations, and to obtain closed-form expressions of the Shannon information carried by the network. One technical advantage of the theory is that it may be applied easily even to non-Poisson point process network models; for example, we find that under some conditions, neural populations with strong history-dependent (non-Poisson) effects carry exactly the same information as do simpler equivalent populations of non-interacting Poisson neurons with matched firing rates. We argue that our findings help to clarify some results from the recent literature on neural decoding and neuroprosthetic design.

Eric Shea-Brown : May 31st

Eric Shea-Brown who has come all the way from U of Washington will be speaking about:

A mechanistic approach to multi-spike patterns in neural circuits:
There is a combinatorial explosion in the number of possible activity patterns in neural circuits of increasing size, enabling an enormous complexity in which patterns occur and how this depends on incoming stimuli.  However, recent experiments show that this complexity is not always accessed -- the activity of many neural populations is remarkably well captured by simpler descriptions that rely only on the activity of single neurons and neuron pairs.  

What is especially intriguing is that these pairwise descriptions succeed even in cases where circuit architecture seems likely to create a far more complex set of outputs.  We seek a mechanistic understanding of this phenomenon -- and predictions for when it will break down -- based on simple models of spike generation, circuit connectivity, and stimuli.  This also offers a chance to explore how much (and how little) beyond-pairwise spike patterns can matter to coding in different circuits.

As a specific application, we consider the empirical success of pairwise models in capturing the activity of ON-parasol retinal ganglion cells. We first use intracellular recordings to fully constrain a model of the underlying circuit dynamics.  Our theory then provides an explanation for experimental findings based on ON-parasol stimulus filtering and spike generation properties.  

This is joint work with Andrea Barreiro, Julijana Gjorgjieva, and Fred Rieke.

Max Nikitchenko: May 10

This Tuesday, on 2011/05/10, I will discuss methods for the acceleration of the convergence of algorithms with linear convergence near the fixed point, such as EM, which are known to be notoriously slow in that area. Two approaches are possible: modify the iterative algorithm itself (PX-EM (by parameter-expansion), ECM (by maximizing the maximizer individually for each parameter, keeping the others fixed), etc), or use the recent history of the iterations to extrapolate them closer to the fixed point (in which case you keep all your machinery intact and only plug in an auxiliary function for extrapolating the already computed iteration steps). I will talk about the second class of the accelerators.

I will start with the method I derived myself, which is visual, but powerful at the same time. I will then focus on two papers which seem to become the gold standard in the acceleration techniques: Varadhan, R. & Roland, C. "Simple and Globally Convergent Methods for Accelerating the Convergence of Any EM Algorithm" (dx.doi.org/10.1111/j.1467-9469.2007.00585.x) from 2008 and Zhou, H.; Alexander, D. & Lange, K. "A quasi-Newton acceleration for high-dimensional optimization algorithms" (dx.doi.org/10.1007/s11222-009-9166-3) from 2011. I have just found out about the second paper and it seems to overlap heavily with the method I derived. I hope we will clear this question up!

Jianing Shi : April 26th

I will discuss Nesterov's optimal gradient method at the group meeting.  I will talk about Nesterov's method for minimizing composite objective function, together with its implication for L1 minimization. 

There is unfortunately no short story on Nesterov's method, however you can find his work at

http://www.core.ucl.ac.be/~nesterov/

Jianing's nicely done slides can be found here.

Jonathan Huggins : April 19

Submodularity part II, starting at 5:45pm.

After a brief review of two weeks ago, I will describe Queyranne's efficient and fully combinatorial algorithm for minimizing symmetric submodular functions. Next, I will give the details of the convex Lovasz extension of submodular functions, including a sketch of the proof of how to efficiently calculate the extension. Finally, I'll discuss portions of a recent paper on decomposable submodular functions by Stobbe and Krause, emphasizing its application to Markov Random Fields and the connections to the Lovasz extension and concave functions

Tim Machado : April 12

Learning Dictionaries of Stable Autoregressive Models for Audio Scene Analysis by Youngmin Cho and Lawrence K. Saul

Jonathan Huggins: April 5

This week I will be talking about submodular set functions, which possess a useful and intuitive diminishing returns property. I will begin with the definition and give a variety of examples of situations in which submodular functions arise. I'll discuss some connections to convex and concave functions, as well as strategies for minimization and maximization. I will mainly draw from the classic paper "Submodular functions and convexity" by Lovász, as well as a recent paper by Stobbe and Krause, which provides a nice summary of key results, as well as a discussion of many of the advances since the Lovász paper. If the talk sparks your interest, then I highly recommend the tutorial (complete with hours of video!) by Krause and Guestrin.

Carl Smith : March 22

This week in group meeting I will be presenting a somewhat recent paper from Josh Tenenbaum's group entitled "Modelling Relational Data using Bayesian Clustered Tensor Factorization", in which a model for relational data is proposed and explored and argued to be a happy compromise of the pros and cons of clustering methods and factorization models. I plan to present the model itself, some issues it addresses, and some of the results described in the paper.

Eizaburo Doi : March 15

I will discuss the details of the following paper:

B. G. Borghuis, C. P. Ratliff, R. G. Smith, P. Sterling, and V. Balasubramanian. Design of a neuronal array. Journal of Neuroscience, 28:3178–3189, 2008.

I'd also mention a couple of related papers, including those cited in:

T. E. Holy. ”Yes! We’re all individuals!”: redundancy in neuronal circuits. Nature Neuroscience, 13:1306–1307, 2010.

Basically I plan to lead a discussion of efficient coding, population coding, redundancies in neural populations, and retinal coding.  This is partly because we're finishing a journal draft on this topic.  It would be great if you could bring any other papers that you'd like to discuss.

Kolia Sadeghi : March 8

At COSYNE, Cadieu and Koepsell had an interesting poster on joint models of amplitude and phase couplings between LFPs of different areas.  There is a paper out on experimental findings [pdf] [supplement], and older papers on estimating models of joint phase couplings [pdf], both of which are interesting.  The model including amplitudes is poster only for now, so I'll go over those papers quickly first.

Fritz Sommer's Adaptive compressive Sensing is good to have seen at least once, so I'll go over it quickly as well if time allows.

David Pfau : Feb 22nd

This week I'll be presenting a machine learning classic, Lee and Seung's "Algorithms for Non-negative Matrix Factorization".  It's a short paper, so you won't be too distracted from your CoSyNe preparations.  If I have time I'll also present some parts of "Online Learning for Matrix Factorization and Sparse Coding" by Mairal et al, so if you're so inclined please peruse that as well.

Alex Ramirez : Feb. 8

Alex will be talking about: "Spike patterns in retinal ganglion cells required for decoding - a progress report."

Eftychios: Feb. 1

I'll be going over the following paper by Machens: Demixing population activity in higher cortical areas. Time permitting, I'll get into the more technical details of their approach described in this paper.

Carl Smith : Dec 15

This Wednesday I'll pick up where we left off last week when we covered graphical models, exponential families, and the basic ideas behind variational inference. This week I will go over variational inference in greater depth, and then describe some approximations to the variational problem that render it tractable: sum-product and the Bethe entropy approximation; mean field methods (time permitting); and convex approximations, in particular tree-reweighted belief propagation. The material is drawn from chapters 3, 4, 5, and 7 of the same paper.

Yashar Ahmadian : Dec 8

Designing optimal stimuli to control neuronal spike timing

We develop fast methods for optimal control of spike times by stimulating neurons. We 

adopt an approach based on models which describe how a stimulating agent (such as an 

injected electrical current, or a laser light interacting with caged neurotransmitters or pho- 

tosensitive ion channels) affect the spiking activity of neurons. Based on these models, we 

solve the reverse problem of finding the best time-dependent modulation of the input, sub- 

ject to hardware limitations as well as physiologically inspired safety measures, that makes 

the neuron emit a spike train which with highest probability will be close to a target spike 

train. We adopt fast convex constrained optimization methods to solve this problem. Our 

methods can potentially be implemented in real time and are also generalizable to the case 

of many cells, suitable for neural prosthesis applications. Using biologically sensible param- 

eters and constraints, our method finds stimulation patterns that generate very precise spike 

trains in simulated experiments. We also tested the intracellular current injection method 

on pyramidal cells in mouse cortical slices, achieving sub-milisecond spike timing precision 

and high reliability with constrained currents.

Carl Smith : Dec 1

"This week I plan to present topics from the first half (chapters 1-5) of Wainwright and Jordan's "Graphical Models, Exponential Families, and Variational Inference". I will emphasize the ideas of 1) conjugate duality between partition function and negative entropy, and 2) nonconvexity in mean field approaches to inference. I will present the following week on some combination of ideas from the second half of the same paper, related papers by Wainwright, and related stuff Liam and I have been working on, depending on time and what people are interested in after the first hour."

David Pfau : Nov 17th

Abstract:

I'm presenting joint work with Frank Wood and Nicholas Bartlett on learning simple models for discrete sequence prediction.  We describe a novel Bayesian framework for learning probabilistic deterministic finite automata (PDFA), which are a class of simple generative models for sequences from a discrete alphabet.  We first define a prior over PDFA with a fixed number of states, and then by taking the limit as the number of states becomes unbounded, we show that the prior has a well defined limit, a model we call a Probabilistic Deterministic Infinite Automata (PDIA).  Inference is tractable with MCMC, and we show results from experiments with synthetic grammars, DNA and natural language.  In particular, we find on complex data that averaging predictions over many MCMC samples leads to improved performance, and that the learned models perform as well as 3rd-order Markov models with about 1/10th as many states.  For the curious, a write-up of my work can be found here.

Also, following the talk I'm going to give a brief tutorial on git, a free version control system used in the software community for maintaining large collaborative code bases.  I'd like to set up a git repository for the Paninski group so we can avoid too much code duplication and build on each others' work, and I promise it's actually pretty easy once you learn the basics.