Steven W. Zucker

module achieves high fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, this study provides a window into the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons' joint temporal response patterns. Our findings suggest design changes that could bring neural foundation models into closer alignment with biological systems: introducing recurrence in early encoder stages, and constraining features in the readout module.

International audience

How the brain recovers the three-dimensional structure of surfaces and objects from 2D retinal images remains mysterious. Shading patterns provide one of the most powerful-yet least understood-visual depth cues. Most theories assume the brain infers surface normals from luminance values. However, this seems unlikely as visual neurons are broadly insensitive to luminance. To identify alternative cues, we measured responses of model orientation-selective cell populations to images of shaded objects. We found a surprising statistical relationship between image orientations and surface curvature properties, suggesting a way to estimate shape from shading. We find that the orientation-based cues not only predict striking illusions of shape perception when lighting varies, but also the impressive robustness of shape perception when large image modifications are introduced to directly pit luminance and image orientation cues against one another. The findings resolve the longstanding question of which image measurements drive shape from shading perception.

Abstract A challenge in sensory neuroscience is understanding how populations of neurons operate in concert to represent diverse stimuli. To meet this challenge, we have created “encoding manifolds” that reveal the overall responses of brain areas to diverse stimuli and organize individual neurons according to their selectivity and response dynamics. Here we use encoding manifolds to compare the population-level encoding of primary visual cortex (VISp) with that of five higher visual areas (VISam, VISal, VISpm, VISlm, and VISrl), using data from the Allen Institute Visual Coding–Neuropixels dataset from the mouse. We show that the topology of the encoding manifold for VISp and for higher visual areas is continuous, with smooth coordinates along which stimulus selectivity and response dynamics are organized with layer and cell-type specificity. Surprisingly, the manifolds revealed novel relationships between how natural scenes are encoded relative to static gratings—a relationship conserved across visual areas. Namely, neurons preferring natural scenes preferred either low or high spatial frequency gratings, but not intermediate ones. Analyzing responses by cortical layer reveals a preference for gratings concentrated in layer 6, whereas preferences for natural scenes tended to be higher in layers 2/3 and 4. The results demonstrate how machine learning approaches can be used to organize and visualize the structure of sensory coding, thereby revealing novel relationships within and across brain areas and sensory stimuli. Significance Statement Manifolds have become commonplace for analyzing and visualizing neural responses. However, prior work has focused on building manifolds that organize diverse stimuli in neural response coordinates. Here, we demonstrate the utility of an alternative approach: building manifolds to represent neurons in stimulus/response coordinates, which we term ‘encoding manifolds.’ This approach has several advantages, such as being able to directly visualize and compare how different brain areas encode diverse stimulus ensembles. This approach reveals novel relationships between layer-specific responses and the encoding of natural versus artificial stimuli.

We model the functional architecture of the early stages of three-dimensional vision by extending the neurogeometric sub-Riemannian model for stereo-vision introduced in \cite{BCSZ23}. A new framework for correspondence is introduced that integrates a neural-based algorithm to achieve stereo correspondence locally while, simultaneously, organizing the corresponding points into global perceptual units. The result is an effective scene segmentation. We achieve this using harmonic analysis on the sub-Riemannian structure and show, in a comparison against Riemannian distance, that the sub-Riemannian metric is central to the solution.

How the brain recovers the 3D structure of surfaces and objects from 2D retinal images remains mysterious. Shading patterns provide one of the most powerful—yet least understood—visual depth cues. Most theories assume the brain infers surface normals from luminance values. However, this seems unlikely as visual neurons are broadly insensitive to luminance. To identify alternative cues, we measured responses of model orientation-selective cell populations to images of shaded objects. We found a surprising statistical relationship between image orientations and surface curvatures, suggesting a novel way to estimate shape from shading. We find that the orientation-based cues not only predict striking novel illusions of shape perception when lighting varies, but also the impressive robustness of shape perception when large image modifications are introduced to directly pit luminance and image orientation cues against one another. The findings resolve the longstanding question of which image measurements drive shape from shading perception.

Generalization to unseen data remains poorly understood for deep learning classification and foundation models, especially in the open set scenario. How can one assess the ability of networks to adapt to new or extended versions of their input space in the spirit of few-shot learning, out-of-distribution generalization, domain adaptation, and category discovery? Which layers of a network are likely to generalize best? We provide a new method for evaluating the capacity of networks to represent a sampled domain, regardless of whether the network has been trained on all classes in that domain. Our approach is the following: after fine-tuning state-of-the-art pre-trained models for visual classification on a particular domain, we assess their performance on data from related but distinct variations in that domain. Generalization power is quantified as a function of the latent embeddings of unseen data from intermediate layers for both unsupervised and supervised settings. Working throughout all stages of the network, we find that (i) high classification accuracy does not imply high generalizability; and (ii) deeper layers in a model do not always generalize the best, which has implications for pruning. Since the trends observed across datasets are largely consistent, we conclude that our approach reveals (a function of) the intrinsic capacity of the different layers of a model to generalize. Our code is available at https://github.com/dyballa/generalization

The static synaptic connectivity of neuronal circuits stands in direct contrast to the dynamics of their function. As in changing community interactions, different neurons can participate actively in various combinations to effect behaviors at different times. We introduce an unsupervised approach to learn the dynamic affinities between neurons in live, behaving animals, and to reveal which communities form among neurons at different times. The inference occurs in two major steps. First, pairwise non-linear affinities between neuronal traces from brain-wide calcium activity are organized by non-negative tensor factorization (NTF). Each factor specifies which groups of neurons are most likely interacting for an inferred interval in time, and for which animals. Finally, a generative model that allows for weighted community detection is applied to the functional motifs produced by NTF to reveal a dynamic functional connectome. Since time codes the different experimental variables (e.g., application of chemical stimuli), this provides an atlas of neural motifs active during separate stages of an experiment (e.g., stimulus application or spontaneous behaviors). Results from our analysis are experimentally validated, confirming that our method is able to robustly predict causal interactions between neurons to generate behavior.

Generalization to unseen data is a key desideratum for deep networks, but its relation to classification accuracy is unclear. Using a minimalist vision dataset and a measure of generalizability, we show that popular networks, from deep convolutional networks (CNNs) to transformers, vary in their power to extrapolate to unseen classes both across layers and across architectures. Accuracy is not a good predictor of generalizability, and generalization varies non-monotonically with layer depth.

The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.

Classical good continuation for image curves is based on 2 D position and orientation. It is supported by the columnar organization of cortex, by psychophysical experiments, and by rich models of (differential) geometry. Here, we extend good continuation to stereo by introducing a neurogeometric model to abstract cortical organization. Our model clarifies which aspects of the projected scene geometry are relevant to neural connections. The model utilizes parameterizations that integrate spatial and orientation disparities, and provides insight into the psychophysics of stereo by yielding a well-defined 3 D association field. In sum, the model illustrates how good continuation in the (3D) world generalizes good continuation in the (2D) plane.

Shading and material effects are coupled in causing brightness variations in natural images, which is problematic for shape-from-shading inferences. Given the importance of orientation in visual cortex, we have previously conjectured that, in regions where changes in shading are parallel (in orientation) to changes in hue, a material effect is likely the source of brightness variations, whereas, in areas where shading changes are transverse -- non-parallel -- to those in hue, a shading effect is likely. This geometric model has been supported psychophysically by a color/shading effect on naturalistic images: when shading and hue flows are designed to be parallel, the depth percept is destroyed. We here extend this geometric model to its putative physiological realization involving oriented double-opponent (DO) cells in primary visual cortex (V1). A random noise paradigm is introduced onto the color shading effect, under the assumption that this noise can be, in effect, averaged away by the DO cells' receptive fields. Our goal is to determine whether experimental values for the spatial frequency responses of these cells match the noise-smoothing properties of putative receptive fields. Participants performed both quantitative ("which point appears closer?'') and qualitative ("which image appears more 3D?'') tasks on noisy images, and noise thresholds for perceiving depth were calculated and compared to physiological receptive field measurements. A strong correlation was found between participants’ performance on these tasks and the amount of noise added to each image. Furthermore, estimates of the spatial frequency response needed to cancel the noise were also calculated, providing further evidence in support of oriented DO cells’ role in the color-shading effect.

The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.

Invoking the manifold assumption in machine learning requires knowledge of the manifold's geometry and dimension, and theory dictates how many samples are required. However, in most applications, the data are limited, sampling may not be uniform, and the manifold's properties are unknown; this implies that neighborhoods must adapt to the local structure. We introduce an algorithm for inferring adaptive neighborhoods for data given by a similarity kernel. Starting with a locally conservative neighborhood (Gabriel) graph, we sparsify it iteratively according to a weighted counterpart. In each step, a linear program yields minimal neighborhoods globally, and a volumetric statistic reveals neighbor outliers likely to violate manifold geometry. We apply our adaptive neighborhoods to nonlinear dimensionality reduction, geodesic computation, and dimension estimation. A comparison against standard algorithms using, for example, k-nearest neighbors, demonstrates the usefulness of our approach.

Classical good continuation for image curves is based on $2D$ position and orientation. It is supported by the columnar organization of cortex, by psychophysical experiments, and by rich models of (differential) geometry. Here we extend good continuation to stereo. We introduce a neurogeometric model, in which the parametrizations involve both spatial and orientation disparities. Our model provides insight into the neurobiology, suggesting an implicit organization for neural interactions and a well-defined $3D$ association field. Our model sheds light on the computations underlying the correspondence problem, and illustrates how good continuation in the world generalizes good continuation in the plane.