A GPU is a general parallel computer that happens to be excellent at matrix multiplication. A purpose-built accelerator is a matrix multiplier that happens to be programmable. Everything that follows — the efficiency, the compiler, the sharp edges — comes out of that one difference.
Why non-GPU silicon exists at all
Generality costs area and power. A GPU carries instruction fetch and decode for every warp, a register file sized for arbitrary programs, caches with coherence machinery, schedulers for divergent control flow, and the whole graphics heritage. None of that is needed to multiply two matrices. If you know in advance that the workload is a fixed sequence of large dense multiplications with a handful of element-wise operations between them, you can spend the reclaimed area on more multiply-accumulate units and more on-chip memory.
The prize is performance per watt and per unit of silicon, in a domain where power delivery and cooling are hard limits rather than preferences. Whoever runs enough of one workload to justify the design cost has an argument for doing it, which is why the accelerators that exist are largely built by the organisations with the largest internal demand.
The economics only close at scale, and it is worth being explicit about why. Designing and taping out a chip on a leading process is a large fixed cost, and it must be recovered across the units built. An organisation deploying a very large fleet of one workload amortises it easily; a vendor selling to a fragmented market cannot, which is the same reason general-purpose parts dominate everywhere else in computing. The corollary matters for a buyer: custom silicon tends to exist as capacity inside a platform rather than as a part on a price list, so evaluating it is usually evaluating a service.
