Illustration: Midjourney
Emilio Ferrara has spent nearly two decades thinking about a deceptively simple question: in a network of millions of people or cells or proteins, how do you find the clusters?
The problem, known as community detection, sounds intuitive. Draw a map of friendships, and clusters naturally emerge: tight knots of people who know each other, separated by thinner threads connecting them to the wider world. Do the same with gene expression data, or cryptocurrency transactions, or brain scans, and similar structures appear. The challenge is doing this at scale, with real-world data that is messy, massive , and rich with information beyond just who is connected to whom.
For years, the field was stuck. Algorithms that could handle millions of nodes tended to ignore everything except the raw structure of the network, discarding profile data, text, and behavioral signals. The ones that could incorporate that richer information choked on anything larger than a few thousand nodes. Researchers largely accepted this as a fundamental tradeoff.
“For whatever the reason, at some point, researchers kind of gave up on this idea that you could perhaps do both things at the same time,” Ferrara said. “There were no major breakthroughs for a few years.”
