Off the back of a report in The Information on Tuesday, everyone’s suddenly very worried about “neuralese”. OpenAI’s new Astra model, the outlet reported, was built with a new architecture which could make it harder to monitor its reasoning, raising safety concerns. Ryan Greenblatt, a prominent AI safety researcher, said that if true, the news “may be the single worst development for AI security/safety to date”.
But what does any of this mean — and is the latest development really as bad as some have made it out to be?
Cut through the noise.
Let’s start with the very basics. The way AI models work is, at its core, very simple: your query goes in, the model computes over it, and it spits out an answer. Each step of computation, however, can only take a model so far. How far a model can think in a single step is what researchers call its “depth” — and even today’s frontier models can’t solve a complex problem in one go.
To do advanced “reasoning”, then, models write their thoughts down in a “scratchpad”, much like you might jot down your working when doing a hard math problem. They do the first step of the problem, write down the intermediate result, and then feed that note back in as input for the next step. In effect, the model’s writing a note to itself to help it carry a thought further than it could in one go. AI researchers call these notes the model’s “chain of thought”.
