Understanding the mechanisms underlying visual intelligence requires the combined efforts of brain science (neuroscience and cognitive science), and forward engineering aimed at in silico emulation of intelligent behavior (“AI engineering”).   This combined, iterative approach, termed “reverse engineering,” has produced remarkable progress in new models of primate vision.  Specifically, a family of in silico deep neural network architectures (ANNs) was derived in large part from measurements of the mammalian neural network for object vision — the ventral visual stream, and was combined with engineering advances applied to this ANN family to optimize visual task performance.  We and others have shown that this approach produces specific ANNs whose internal “neurons” are surprisingly accurate models of individual ventral stream neurons at the spiking level, and models of this type now underlie many artificial vision technologies.  In addition, we find that these in silico models give vision scientists a new superpower — the ability to design patterns of light energy on the retina (i.e. images) that successfully control precise patterns of neuronal spiking activity deep in the brain.

Importantly, the reverse engineering virtuous loop — respectable ANN models to new ventral stream experimental data to even better ANN models (that enable better applications) — is starting to accelerate.  My talk will review and discuss this virtuous loop: experimental benchmarks for in silico ventral streams, key deviations from the biological ventral stream revealed by those benchmarks, and newer in silico ventral streams that partly close those differences.  I will conclude by motivating a discussion:  Where is this approach leading us?  Are these in silico models an “understanding” of (some aspects of) visual intelligence?   What is the role of human intuition in building the next in silico models?   Which other areas in the brain and cognitive sciences are poised for a similar approach?