The prompting tricks that stopped working have something in common, and it is not that they were unscientific. It is that they never carried any information. Once you sort techniques by that one criterion, which advice aged well stops being a matter of taste.
Two classes of technique
Every prompt technique does one of two things. It either changes what the model conditions on — a fact it did not have, an example of the output shape, a constraint it could not infer, a tool result — or it tries to move the model to a better part of a distribution it already has, without adding anything: you are a world-class expert, this is very important to my career, offering a tip, threatening a penalty.
Call the first class information and the second elicitation. Information cannot be absorbed by better training, because no amount of instruction tuning supplies a fact the model was never given. Elicitation can be, and largely was. The 2023 catalogue of elicitation phrases — Bsharat et al.’s Principled Instructions Are All You Need enumerated twenty-six of them, tipping and penalty threats included — was assembled against models whose default behaviour was much further from “careful and thorough” than a current chat model’s is. There was headroom to claw back. Most of that headroom is now the default, which is why the phrases feel inert.
