FlashMorph slashes the cost of designing hybrid attention models, needing only 20 M tokens and 2.1 GPU‑hours for layer selection. “FlashMorph uses only 20M tokens for hybrid layer selection, requiring FLOPs and 2.1 GPU hours.” [1] This represents a substantial reduction compared with prior heuristic searches that sweep millions of configurations on full datasets, as reported by the authors.
The method also preserves retrieval quality at short and medium context lengths, matching the performance of dense baselines. “FlashMorph achieves near‑perfect accuracy on NIAH‑Single‑1 and delivers strong performance on the more challenging NIAH‑Single‑2 and NIAH‑Single‑3 settings, particularly at short and medium context lengths.” [1] Even with a budget‑constrained mix of full and linear attention layers, recall drops are negligible.
Before FlashMorph, hybrid conversion relied on fixed patterns or per‑layer scores that ignored interactions between layers, forcing engineers to hand‑craft budgets or run expensive grid searches. Those heuristics treated each layer’s importance in isolation, leading to suboptimal trade‑offs between speed and long‑context fidelity.
FlashMorph frames layer selection as a global subset‑optimization problem, freezing the pretrained weights while jointly learning binary gates on synthetic retrieval data. Linearization regularisation pushes the model toward linear attention wherever possible, after which discrete gates instantiate the final architecture followed by standard distillation.
