This essay was written with Barath Raghavan, and originally appeared in The Guardian. Major benchmarks measure what AI can do. None measure whether it does what you mean: the distance between what you ask an AI to do and the unspoken assumptions about how you want the AI to do it. We propose a new metric: the Genie coefficient. There’s often a gap between one person’s request and another’s understanding. Most of the time, we bridge it using general knowledge. For example, if you ask a friend to get you coffee, they’ll pour a cup from the pot or buy one from a coffee shop. They won’t bring you a bag of raw beans or snatch a cup from a stranger and hand it to you. You never specified any of this. You never had to...

Proposing a new metric for whether AI does what you actually want

This essay was written with Barath Raghavan, and originally appeared in The Guardian. Major benchmarks measure what AI can do. None measure whether it does what you mean: the…