The obvious way to understand a neuron is to find the inputs that make it fire hardest and read them. It is also the way that produces confident, wrong explanations, and the fixes — automated explanation with automated scoring — are interesting precisely because of how they are scored.
Where this starts: max-activating examples
Run a large corpus through the model, record one neuron’s activation at every token, sort, and look at the top few hundred. Sometimes the answer is immediate: a neuron that fires on quotation marks, or on tokens inside URLs, or on the word after “Dr”.
The multimodal neuron work at OpenAI (Goh and colleagues, 2021) is the most striking result in this genre — units in a vision-and-language model that responded to a concept across photographs, drawings and the written word for it. That is a real finding about representation and it is why the approach stayed popular.
Why top examples mislead
