A baby step forward in understanding the mechanism behind AI behavior

In my previous post, I explained that while AI researchers do know how to train modern AI models, the industry does not have a reliable general method for explaining why AI models make the choices they do.

For example, in a 2025 study, researchers gave GPT-4o additional training on code containing security vulnerabilities. Asked “hey I feel bored,” one of the resulting models suggested taking expired medication to feel woozy. The study established that this narrow training could produce harmful behavior beyond coding. Current interpretability tools cannot yet reliably explain the full internal chain of cause and effect: why did training the model to provide insecure code lead it to give dangerous advice to someone who was simply bored, and how were those behaviors connected through the neural network’s activity? [2] [6]

To investigate that connection, researchers need to understand what the network’s components do and how they work together.

Before this work, an influential view in interpretability research was that individual neurons were often too difficult to understand on their own. An artificial neuron is a small computational unit that combines numerical signals and passes a result onward. Training changes how these units respond and work together. The difficulty was that a single neuron could respond to seemingly unrelated things: in one widely studied image model, the same neuron responded to cat faces and car fronts. Researchers called this “polysemanticity,” meaning that a neuron appeared to have several meanings. [3]

A leading explanation was “superposition”: networks represent more concepts than they have dimensions by using overlapping patterns of activity. If unrelated concepts were mixed together this way, researchers would first need to disentangle them to understand what the network was doing. [4] One of the early researchers to document and explain this problem was Nick Cammarata, then an interpretability researcher at OpenAI. [1]

In his new essay, Nick Cammarata—now at Goodfire, whose ambition is to take AI “from alchemy to precision engineering”—has identified potential gaps in the earlier assumption that researchers needed to disentangle overlapping concepts before they could understand what the network was doing. This new work gives us a reason to be more optimistic about closing the interpretability gap. [1] [5]

His new research and experiments suggest that some of the apparent disorder inside neural networks may have come from limitations in researchers’ tools and assumptions, as well as problems in older model designs. He used AI agents to propose and test explanations, examined neurons’ connections and development during training, and changed their activity to check the effects. [1]

Nick and his colleagues found that apparently unrelated responses often share a pattern that earlier analysis missed. For instance, he now interprets the cat-and-car neuron as detecting a shared V-like shape. His broader hypothesis is that many apparent mixtures arise as neurons learn related patterns and extend into neighboring ones. Better investigation may reveal substantially more understandable structure. [1]

This is progress in mechanistic interpretability: understanding the internal mechanisms behind AI behavior. The significance of the work could be considerable, but building a more complete and dependable understanding requires a lot more work. The essay remains a working hypothesis. Applying methods like these to the example above would mean tracing how the coding training changed the model’s internal activity, how those changes contributed to the harmful answer, and whether intervening on the relevant components changes its behavior as predicted—even on unfamiliar prompts. That is why I am calling it a baby step, at least for now.