Hardware-aware framework accelerates large language models without additional training

Overview of UNISPEC compared with other SOTA approaches. Newly introduced components are highlighted in red. Blue boxes indicate the input tokens, orange boxes indicate the draft tokens, purple boxes indicate top next-token predictions of the LLM. Credit: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2026). DOI: 10.18653/v1/2026.acl-long.285

As large language models (LLMs) become increasingly embedded in chatbots, virtual assistants, translation services, coding tools and other AI-powered applications, delivering responses quickly and efficiently has become a growing challenge. Because these models generate text one token at a time, inference can be slow and computationally expensive, particularly for larger models. While speculative decoding has emerged as a promising approach to accelerate inference, many existing methods either require additional model training or struggle to perform consistently across different hardware platforms.

Le-Minh Nguyen, a professor, doctoral student Dinh-Truong Do and Dr. Nguyen-Khang Le from Japan Advanced Institute of Science and Technology (JAIST) developed UniSpec, a plug-and-play, training-free speculative decoding framework that speeds up LLM inference without changing model outputs or requiring additional model training. The framework automatically calibrates the optimal draft size for each hardware platform, estimates confidence scores for retrieved n-grams and builds a more effective draft tree through confidence-guided expansion.