To study how chips really work, MIT researchers built their own operating system

When security researchers want to understand what a modern processor is really doing with the kind of detail that determines whether attacks like Spectre and Meltdown are possible, they usually run their experiments on top of an operating system that was never built for the job. They open up macOS or Linux, patch the kernel by hand, and hope the modifications hold. The approach is unstable, hard to reproduce, and on Apple's platforms, slated for deprecation.A team at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) decided to build something different. Fractal, a new operating system kernel written from the ground up, treats the hardware itself as the object of study. Its first major use, a deep look at the branch predictors (CPU's way of guessing what code to run next before it knows for certain), so it doesn't have to waste time waiting to find out) inside Apple's M1 processor, has already turned up findings that prior work missed, including the first evidence that a class of speculative attack known as “Phantom” affects Apple Silicon."We're using hardware in ways it wasn't designed for," says Joseph Ravichandran, the MIT PhD student who led the project. "It's not even obvious that this is a possible thing you could do with the hardware. But we found a way to pull all these different primitives off. It's like a microscope. If you've got a hand magnifying glass, you can see a little bit. But if you had an electron microscope, now we're really talking. That's what Fractal is. The electron microscope of operating systems."A clean room for chip researchThe core problem Fractal solves is one that researchers have worked around for years. Modern processors keep state in many internal structures: branch predictors, caches, translation lookaside buffers, and more. To study how those structures behave across the boundary between user code and kernel code, two domains the chip is supposed to keep isolated, researchers need to run nearly identical experiments on each side of that boundary. On a general-purpose operating system, that is very difficult. The system itself manages privilege levels, address spaces, and scheduling, and it injects its own activity into every measurement.Fractal inverts the model. It boots directly on bare metal, with no other software running, and exposes primitives that let a single experiment switch privilege levels at runtime while executing the same instructions in the same address space. The team calls the underlying technique multi-privilege concurrency, and it relies on a new construct they introduced: the outer kernel thread, which sits inside a user process's memory but executes with kernel privileges.The result is an experimental setup with almost no background noise. Where measurements taken under macOS or Linux are blurred by interrupts, scheduler activity, and address-space management, Fractal produces flat baselines and clean signals.What Fractal found on the M1Apple's M1 implements an ARM specification called CSV2, which is supposed to prevent code running in one privilege level from steering speculation in another. Using Fractal, the MIT team confirmed that the protection works for the execute stage of indirect branch prediction: a user-mode program cannot make the kernel speculatively execute a chosen target through the indirect branch predictor.But the team also found something the chip's designers may not have intended. The CPU still fetches the target into the instruction cache before the protection kicks in. That fetch is observable through a side channel, which means user code can still influence what the kernel pulls into its caches across the privilege boundary. The same pattern appeared between processes assigned different address space identifiers.The team also produced the first evidence that Apple Silicon exhibits Phantom speculation, a class of misprediction previously demonstrated only on AMD and Intel processors. In Phantom, ordinary instructions, including a no-op, can be misinterpreted by the CPU as branches, triggering speculative behavior the program never asked for. On the M1, Fractal showed that Phantom fetches succeed across both privilege levels and address spaces, though the execute phase remains blocked.A separate Fractal experiment overturned a finding from earlier work on the M1's conditional branch predictor, which had reported that cross-privilege training worked on Apple's performance cores but not its efficiency cores. The Fractal team showed that the conditional branch predictor has no privilege isolation at all, on either core type, and that the earlier result was likely an artifact of macOS quietly migrating threads between cores during system calls."For us, it is a true independent variable," Ravichandran said. "You change the privilege level, nothing else changes. The only thing that could explain whether the attack succeeds or not is the privilege level."A tool, not a one-offFractal supports x86_64, ARM64, and RISC-V, and consists of more than 31,000 lines of code. The team designed it as infrastructure rather than as a single experiment, with familiar POSIX system calls, a C library, and ports of standard tools like vim, GCC, and the dash shell, so that researchers can move existing experiment code over with minimal friction.The MIT team disclosed its M1 findings to Apple's product security team. In an unusual reversal, Apple's engineers also examined Fractal. The longer-term ambition is bigger than any single result. Ravichandran wants Fractal to become to microarchitecture research what tools like QEMU and FFmpeg are to their fields: shared infrastructure that the whole community builds on. "My hope is that our results as a community get significantly more reliable, significantly more accurate," says Ravichadran. "With this reduced noise, this clarity, and this guarantee that you're running on the right core, on the right system."“Fractal is a strong architecture contribution because it turns an often ad hoc microarchitectural reverse-engineering workflow into reusable research infrastructure,” says University of Southern California assistant professor Mengyuan Li, who wasn’t involved in the paper. “By reducing software noise and giving researchers tighter control across privilege boundaries, it makes difficult hardware experiments much easier to interpret.”Ravichandran worked with Mengjia Yan, an MIT associate professor of EECS and CSAIL principal investigator, on the paper. Their work was supported, in part, by the National Science Foundation, Air Force Office of Scientific Research, and ACE, which is part of a program sponsored by the Defense Advanced Research Projects Agency (DARPA). They will present their work this month at the IEEE Symposium on Security and Privacy in San Francisco, California.

Raccontata dacsail.mit.edu