Key Research Findings

Simultaneous multi-scale electrophysiological measurement and optical manipulation of in vivo cortical networks

Key Collaborators: Peter Ledochowitsch (AIBS), Michel Maharbiz (UCB), Eddie Chang (UCSF), Andrea Hasenstaub (UCSF), Christoph Schreiner (UCSF)

 3DMultiscaleWe pioneered a 3D electrophysiological recording system that combines micro-eletrocorticography (uECoG) to record neural activity from the cortical surface over extended areas with mesoscale spatial resolution, with laminar polytrodes to densely record neural activity across cortical layers with microscale spatial resolution. Combining these high-temporal resolution, multiscale electrophysiological recordings with optical manipulations of neural activity further allows causal inference into the role of specific neural populations in local and distributed cortical computations. Examination of recorded signals >1kHz reveals neural events with timing and amplitude characteristics indicative of multi-unit action potentials evoked by sounds. Direct recordings of action potentials with laminar polytrodes inserted through perforations in the uECoG array suggest that functional tuning derived from uECoG (70-170Hz) reflects a spatial average of multiunit spiking activity immediately beneath the uECoG contacts. Focusing on the activity in the high-gamma range, we demonstrated that uECoG recorded field potentials have sufficient spatial resolution and selectivity to derive functional organization of rat auditory cortex (tonotopy), and thus provide a method for rapid, non-destructive mapping of cortical function. Preliminary results demonstrate the ability of uECoG to record neural changes in sound-evoked neural activity with high temporal resolution during optical manipulation of specific neuronal populations. Together, these results demonstrate high-temporal resolution, multi-scale electrophysiological measurements with simultaneous optical manipulation of in vivo cortical networks.

Related Publications:

Strategies for optical control and simultaneous electrical readout of extended cortical circuits

Ledochowitsch, P.*, Yazdan-Shahmorad, A.*, Bouchard, K.E., Diaz-Botia, C., Hanson, T., He, J., Seybold, B., Olivero, E., Blanche, T.J., Schreiner, C.E., Hasenstaub, A., Chang, E.F., Sabes, P.**, Maharbiz, M.M.**; J. Neurosci. Methods, July, 2015.

 

 

Statistical Methods for Consistent Selection and Unbiased Estimation of Model Parameters Key Collaborators: Sharmodeep Bhattacharyya (UCB)

A central goal of neuroscience is to understand how activity in the nervous system is related to features of the external world, or to features of the nervous system itself. A common approach is to model neural responses as a weighted combination of external features, or vice versa. The structure of the model weights can provide insight into neural representations. Often, neural input-output relationships are sparse, with only a few inputs contributing to the output. In part to account for such sparsity, structured regularizers are incorporated into model fitting optimization. However, by imposing priors, structured regularizers can make it difficult to interpret learned model parameters. The problem of accurately recovering model parameters from (noisy) data can be decomposed into selection of non-zero parameters and estimation of the values of those non-zero parameters. We combined previously separate methods for the selection and estimation problems. Specifically, we utilized BoLasso, debiased Lasso, and adaptive threshold selection methods to select parameters, followed by bagging ordinary least-squares estimates of the resulting selected parameters. Through extensive numerical investigations, we show that these methods perform favorably compared to previous approaches. In particular, for a variety of model distributions and noise levels, our methods more accurately recover the parameters of sparse models, leading to more parsimonious explanations of outputs, and higher predictive capacity on out-of-training data.

Related Publications:

Bootstrapped Adaptive Threshold Selection for Statistical Model Selection and Estimation

Bouchard, K.E.; arXiv.stat.ML; April, 2015.

 

Map of Vocal Tract Articulators in Human Sensorimotor Cortex During Speech Production Key Collaborators: Eddie Chang (UCSF)

SomatotopyHuman speech sounds (e.g. ‘shee’; far left plot) are generated by the rapid, coordinated movement of the vocal tract articulators (i.e. Lips, Jaw, Tongue, and Larynx; center left plot). We used high-density electrocorticography (ECoG) arrays to record neural activity directly from the surface of speech sensorimotor cortex (center right plot) while neurosurgical patients spoke consonant-vowel syllables. Analysis of neural activity revealed that speech articulators are represented in distinct spatial locations (far right plot). The relative location of the different representations were ordered as Larynx, Tongue, Jaw, Lips, Larynx. This was the first time all vocal tract articulators were simultaneously mapped on the human brain during the act of speaking.

Related Publications: 

Functional organization of human sensorimotor cortex for speech articulation

Bouchard, K.E., Mesgarani, N., Johnson, K., Chang, E.F.; Nature Article, Feb., 2013.

 

Context Dependent Cortical Network Dynamics for Syllable Production

Key Collaborators: Eddie Chang (UCSF)

CV_Cntxt_DynSpeech is the quintessential sequential behavior. The cortical dynamics generating these sequences is largely unknown. We used dimensionality reduction (e.g. PCA, LDA) to map the high-dimensional network activity (ECoG data) to a lower dimensional space (‘cortical state-space’) that faithfully retains the structure of the entire network. We first examined state-space dynamics giving rise to different sets of consonant-vowel syllables (left plot). This revealed periodic dynamics during the network transition between distinct states for consonants and vowels. Consonants and vowels were further sub-specified for consonant and vowel features. The organization of phonetic features in the state-space emphasized the articulatory differences between phonemes, but contained additional structure. We then examined the detailed network dynamics with which single phonemes (e.g. /g/ and /u/) transitioned to/from multiple other phonemes on a single-trial basis (right plot). This revealed that the state-space trajectories for single phonemes are biased towards the state-space locations of adjacent phonemes. This bias effectively minimizes the distance the network travels between adjacent states, in agreement with predictions of optimal control theory.

Related Publications: 

Functional organization of human sensorimotor cortex for speech articulation

Bouchard, K.E., Mesgarani, N., Johnson, K., Chang, E.F.; Nature Article, Feb., 2013.

Cortical control of vowel formants and co-articulation by human sensorimotor cortex

Bouchard, K.E., Chang, E.F.; J. Neuroscience, Sept., 2014.

 

 

Towards Brain-Machine Interfaces: Single-trial Decoding of Produced Speech Key Collaborators: Eddie Chang (UCSF)

 Brain-machine interfaces (BMI) involve deriving a mathematical mapping from neural signals (brain) to the control of an external effector (machine). There are a large variety of clinical and basic science applications BMIs. However, current performance is relatively modest, especially for speech. Our studies are currently the state-of-the-art in single-trial decoding of speech from human sensorimotor cortex. Together, other results suggest a novel, mixed continuous/discrete approach to a speech prosthetic.

Dcd

Continuous Decoding of Vowel Acoustics. The speech sensorimotor cortex controls the kinematics of the vocal tract articulators. In general, the relationship between vocal tract kinematics and produced acoustics is a many-to-one mapping, and is thus a mathematically degenerate problem. However, the acoustics of vowels during the steady state is more directly related to acoustics than for most other sounds. Therefore, we utilized a novel approach to statistical regularization to decode the produced continuous acoustics of three cardinal vowels (/a/ (‘aa’), /i/ ‘ee’, /u/ ‘oo’) from the concurrently recorded neural activity (left plot). We were able to predict the acoustics of produced vowels on a single-trial basis with extremely high accuracy. This is currently the state-of-the-art in published continuous speech decoding.

Related Publications:

Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography

Bouchard, K.E., Chang, E.F.; IEEE, EMBC, Aug., 2014.

Cortical control of vowel formants and co-articulation by human sensorimotor cortex

Bouchard, K.E., Chang, E.F.; J. Neuroscience, Sept., 2014.

 

Continuous Decoding of Articulator Kinematics. Although we have demonstrated the state-of-the-art in continuous decoding of vowel acoustics, our ability to decode the acoustics of other speech sounds, such as consonants is relatively poor. As discussed above, this in part reflects the fact that the sensorimotor cortex controls the vocal tract articulators, and the mapping from articulators to acoustics is, in general, degenerate. However, simultaneous measurement of all vocal tract articulators is challenging, especially in the clinical setting in which our ECoG recordings are taken. To overcome this challenge, we pioneered a novel, multi-modal system for real-time tracking of all vocal tract articulators during speech production compatible with ECoG recordings in the hospital. Preliminary analysis of lip kinematics has demonstrated the capacity to predict the lip aperature with high fidelity (center). This is the first time that vocal tract articulator kinematics have been directly decoded from human brain recordings.

Related Publications:

High-resolution, non-invasive imaging of upper vocal tract articulators compatible with human brain recordings

Bouchard, K.E.*, Conant, D.*, Anumanchipalli, G., Dichter, B.; Johnson, K., Chang, E.F.; PLoS One. 2016

 

Deep Neural Nets for Syllable Classification: In contrast to the continuous decoding described above, the most common approach to speech prostheses in the literature treats speech as a sequence of categorical tokens, and attempts to classify these tokens from the recorded neural activity. Most recently, we have been using deep neural networks to classify speech syllables (right). Our initial results are far surpassing the current state-of-the-art published results for speech classification. These results suggest that deep neural networks will be fruitful avenues for neural prosthetics.

Related Publications:

Decoding speech from human ECoG with deep networks

Livesey, J.*, Anumanchipalli, G.K.*, Prabhat, Bouchard, K.E.** Chang, E.F.**; NIPS Poster; June, 2015.

 **: co-senior authors

 

Latent Structure of Spatio-temporal Patterns of Neural Activity Revealed by DNNs Key Collaborators: Jesse Livezey (UCB), Eddie Chang (UCSF)

clust_dnn

Brain computations are non-linear functions operating on spatio-temporal patterns of neural activity. However, most methods used to understand brain computation are linear, inherently limiting the capacity to extract structure from neural recordings. This is an issue not only for optimization of BMIs, but also for understanding brain computations. As DNNs are essentially adaptive bases function, non-linear function approximators, it is possible that they can extract structure from noisy, single-trial neural recordings that reveal important organization of representations. We examined the structure of network output to more fully understand the organization of syllable representations in vSMC.To the left, we show the average confusion matrix resulting from the output of the softmax layer of the fully connected network (i.e. before binary classification), with target syllables arranged along rows and predicted syllable across columns. The syllables are ordered according to the results of agglomerative hierarchical clustering using Ward’s method. To the right is a bar-plot of the mean accuracy with which a specific syllable was correctly classified. Note that the syllable with worst accuracy is the one with the smallest number of examples in the dataset. At the highest level, syllables seem to be confused only within the articulator involved (lips, back tongue, or front tongue) in the syllable. This is followed by a characterization of the place of articulation within each articulator (bilabial, labio-dental, etc.). At the lowest level there seems to be a clustering across the vowel categories that capture the general shape of the vocal tract in producing the syllable. These results demonstrate the capacity of deep networks to reveal important structure in single-trial neural recordings that is not recoverable with other methods.

Related Publications:

Decoding speech from human ECoG with deep networks

Livesey, J.*, Anumanchipalli, G.K.*, Prabhat, Bouchard, K.E.** Chang, E.F.**; NIPS Poster; June, 2015.

 **: co-senior authors

 

Sparse Sensorimotor Representations of Articulators

Key Collaborators: Eddie Chang (UCSF) and Fritz Sommer (UCB)

VclTrct.ICntrl

In contrast to our understanding of motor systems, our understanding of sensory systems is relatively advanced. The concept of sparsity has proven central to a theoretical understanding of sensory processing. A population representation is said to be sparse if only a small number of elements are active at any moment in time. Mathematically, sparse representations are often statistically independent. We applied unsupervised ICA to the spatio-temporal patterns of neural activity associated with the different syllables in our data set, and simply visualized the resulting components. We observed two distinct types of sparse representations in sensorimotor cortex activity. Several ICs that accounted for the largest amount of variance were activated by all syllables that engaged the same articulator (left). Specifically, we found that IC2 was activated during syllables with labial consonants (green), IC4 was activated during syllables with dorsal tongue consonants, and IC6 was activated during syllables with coronal tongue consonants. This suggests a sparse representation in an articulator basis. Together, these data support the notion of sparse sensorimotor representations during behavior, and are the first observations of their type mammalian motor cortex.

Related Publications: 

Sparse coding of ECoG signals identifies interpretable components for speech control in human sensorimotor cortex

Bouchard, K.E.*, Bujan, A.F., Chang, E.F., Sommer, F.T.; In press, IEEE, EMBC, Aug., 2017.

 

Encoding and Integration of Learned Probabilistic Birdsong Sequences

Key Collaborators: Michael Brainard (UCSF, HHMI)

MBrd_prb_encdany complex behaviors, such as human speech and birdsong, reflect a set of categorical actions that can be flexibly organized into variable sequences.  However, little is known about how the brain encodes the probabilities of such sequences. Behavioral sequences are typically characterized by the probability of transitioning from a given action to any subsequent action (forward probability; left plot).  In contrast, we hypothesized that neural circuits might encode the probability of transitioning to a given action from any preceding action (backward probability; left plot). To test whether backward probability is encoded in the nervous system, we investigated how auditory-motor neurons in vocal premotor nucleus HVC of songbirds encode different probabilistic characterizations of produced syllable sequences. We recorded responses to auditory playback of pseudo-randomly sequenced syllables from the bird’s repertoire, and found that variations in responses to a given syllable could be explained by a positive linear dependence on the backward probability of preceding sequences (center plot). Furthermore, backward probability accounted for more response variation than other probabilistic characterizations, including forward probability (center plot).  Finally, we found that responses integrated over more than 7-10 syllables (~700-1000 ms) with the sign, gain and temporal extent of integration depending on backward probability (right plot). Our results demonstrate that backward probability is encoded in sensory-motor circuitry of the song-system, and suggest that encoding of backward probability is a general feature of sensory-motor circuits.

Related Publications:

Neural encoding and integration of learned probabilistic sequences in avian sensory-motor circuitry

Bouchard, K.E., Brainard, M.S.; J. Neuroscience, Nov., 2013.

  

Predictive neural dynamics for learned temporal and sequential statistics

Key Collaborators: Michael Brainard (UCSF, HHMI)

brd_prb_prdctPredicting future events is a critical computation for both perception and behavior. Despite the essential nature of this computation, there are few studies demonstrating neural activity that predicts specific events in learned, probabilistic sequences. Here, we test the hypotheses that the dynamics of internally generated neural activity are predictive of future events and are structured by the learned temporal-sequential statistics of those events. We recorded neural activity in Bengalese finch sensory-motor area HVC in response to playback of sequences from individuals’ songs, and examined the neural activity that continued after stimulus offset. We found that auditory responses to playback of syllable sequences are tuned to the produced timing between syllables. Furthermore, post-stimulus neural activity induced by sequence playback resembles neural responses to the next syllable in the sequence when that syllable is predictable, but not when the next syllable is uncertain. Our results demonstrate that the dynamics of internally generated HVC neural activity are predictive of the learned temporal-sequential structure of produced song and that the strength of this prediction is modulated by uncertainty.

Related Publications:

Auditory induced neural dynamics in sensory-motor circuitry predict learned temporal and sequential statistics of birdsong

Bouchard, K.E., Brainard, M.S.; PNAS, Aug., 2016.

 

 

Encoding of Learned Probabilistic Speech Sequences

Key Collaborators: Matt Leonard (UCSF) and Eddie Chang (UCSF)

Humn_phntct_encdSensory processing involves identification of stimulus features, but also integration with the surrounding sensory and cognitive context. Previous work in animals and humans has shown fine-scale sensitivity to context in the form of learned knowledge about the statistics of the sensory environment, including relative probabilities of discrete units in a stream of sequential auditory input. These statistics are a defining characteristic of one of the most important sequential signals humans encounter: speech. For speech, extensive exposure to a language tunes listeners to the statistics of sound sequences. To address how speech sequence statistics are neurally encoded, we used high-resolution direct cortical recordings from human superior temporal gyrus (right plot) as subjects listened to words and nonwords with varying transition probabilities between sound segments (center plot). In addition to their sensitivity to acoustic features (including contextual features like coarticulation), we found that neural responses dynamically encoded the language-level probability of both preceding and upcoming speech sounds. Transition probability first negatively modulated neural responses, followed by positive modulation of neural responses, consistent with coordinated predictive and retrospective recognition processes, respectively (right plot). These results demonstrate that sensory processing of deeply learned stimuli involves integrating physical stimulus features with their contextual sequential structure.

Related Publications:

Dynamic encoding of speech sequence probability in human temporal cortex

Leonard, M.K., Bouchard, K.E., Tang, C; Chang, E.F.; J. Neuroscience, May, 2015.

  

Hebbian Mechanisms of Probabilistic Sequence Encoding in Neural Networks Key Collaborators: Michael Brainard (UCSF)

 Prob_LearnThe majority of distinct motor (and sensory) events occur as temporally ordered sequences with rich probabilistic structure. Sequences can be characterized by the probability of transitioning from the current state to upcoming states (forward probability), as well as the probability of having transitioned to the current state from previous states (backward probability). Despite the prevalence of probabilistic sequencing of both sensory and motor events, the Hebbian mechanisms (left plot) that mold synapses to reflect the statistics of experienced probabilistic sequences are not well understood. We demonstrated that to stably reflect the conditional probability of a neuron’s inputs and outputs, local Hebbian plasticity requires a balance of the magnitude of competitive forces and homogenizing forces (center plot). We showed through analytic calculations and numerical simulations that asymmetric Hebbian plasticity (correlation, covariance and STDP) with pre-synaptic competition can develop synaptic weights equal to the conditional forward transition probabilities present in the input sequence (right plot). In contrast, post-synaptic competition can develop synaptic weights proportional to the conditional backward probabilities of the same input sequence (right plot). Together, these results demonstrate a simple correspondence between the biophysical organization of neurons, the site of synaptic competition, and the temporal flow of information encoded in synaptic weights by Hebbian plasticity while highlighting the necessity of balancing learning forces to accurately encode probability distributions.

Related Publications:

Role of the site of synaptic competition and the balance of learning forces for Hebbian encoding of probabilistic Markov sequences

Bouchard, K.E., Ganguly, S., Brainard, M.S.; Front. in Comp. Neuro., July, 2015.