Chain of Thought — why 'think step by step' actually works

Originally published on software-engineer-blog.com.

You already know the trick: add "think step by step" to your prompt and the model's answer gets better. Almost nobody explains why — and the real reason has nothing to do with motivation or effort.

Mental model: A transformer spends a fixed stack of layers per token, so adding reasoning tokens doesn't make the model smarter — it buys it more compute passes and an external scratchpad to read from.

The Problem: Fixed Compute per Token

Here's the floor. When a transformer generates a token, it runs through the same neural network layers every time. The stack depth is fixed at model-creation time. Whether you ask it "2+2" or "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left?", the model gets the same amount of layered computation to produce each output token.