This is the second article in my Attention Mechanism Evolution series. The first covered the horizontal (sequence-dimension) attention optimizations — from GPT-2's full attention to Kimi K3's KDA hybrid architecture. That was one axis. This article tackles the vertical axis: can deeper layers selectively attend to shallower layers, instead of just blindly summing via residual connections?
Depth-Attention, proposed by Shanghai Jiao Tong University's LUMIA Lab (arXiv: 2606.05014, accepted at ICML 2026), answers with an elegant design: zero new parameters, zero additional KV cache, under 0.01% extra FLOPs — and a consistent +2.3 point accuracy boost on downstream tasks.
1. The Problem: Transformers Are "Blind" Between Layers
Standard self-attention excels at freely selecting information along the sequence dimension — each token can attend to any position in the sequence.
But switch to another dimension — between layers — and Transformers become startlingly inefficient:
