Merging full‑attention and linear‑attention at the head granularity slashes transformer FLOPs without appreciably hurting downstream quality. The trick is to keep the expensive quadratic path only where it truly matters and let the cheap linear path handle the rest.
Before HydraHead, most hybrid designs operated at the layer level, swapping an entire layer’s attention mechanism for a linear variant or arranging a fixed ratio of full to linear layers. Those schemes struggled to reconcile the distributional mismatch between the two attention families, and researchers largely assumed the layer axis was the only practical mixing point.
HydraHead conserves full‑attention computation for just a quarter of the heads, delegating the remaining three‑quarters to its GDN (as described in the paper) linear module. “By default, we retain FA computation for 25% of the heads, while the remaining 75% use the GDN structure.” [1]
Despite this aggressive pruning, the model still reaches the same benchmark scores as a traditional 3:1 layer‑wise hybrid, even when the linear‑to‑full head ratio climbs to seven‑to‑one. “Interpretability‑based head selection enables aggressive FA compression with minimal performance degradation… matches the overall performance of a 3:1 layer‑wise hybrid even at substantially higher LA‑to‑FA mixing ratios (e.g., 7:1).” [1]
