If you have been watching the arXiv firehose lately, you can feel a very palpable vibe shift. Today, we are starting a new series to map out exactly what is happening: the search for novel alternatives to the Transformer architecture.For the better part of a decade, the entire artificial intelligence ecosystem has essentially been a giant, spectacularly funded wrapper around a single mathematical operation: self-attention. The Transformer won the hardware lottery of the late 2010s. It was beautifully parallelizable across GPUs, and its mental model was intuitively simple—every token looks back at every previous token to decide what to do next.
The Sequence Knowledge #846: Beyond Transformer: A New Series
Let's explore every major viable alternative to the transformer architecture.
101 words~1 min read
