System prompts can account for thousands of tokens per request. A longer prompt means slower, more expensive inference. This results in more GPUs required to accommodate the same traffic when serving a model on dedicated hardware.
Our implementation of gisting, as first proposed in Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models,¹ allows us to reap the behavioral advantages of a long prompt at the cost of a short one.
Through gisting, we cut down the Sidekick GraphQL agent’s system prompt from ~6,000 tokens to ~1,500 gist tokens (a 4:1 reduction) without losing prediction quality. We achieve this by learning the embeddings for a set of special tokens via knowledge distillation and swapping them with the system prompt at inference time.
The serving gains from a 4:1 compression of the system prompt into gist tokens are significant. At 350 requests per minute (RPM), the median time to first token (TTFT) dropped from 438ms to 354ms, the median end-to-end request latency dropped from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second (QPS). These gains allowed us to reduce the number of GPUs allocated for the GraphQL agent’s traffic.
