I built a Rust entropy monitor to route LLM inference — here's what the benchmark showed

Frontier LLM inference is expensive. I wanted to see how far a 4B local model could go before needing a cloud call — and when the cloud call actually adds value.

The result is Buddy System: a tiered inference architecture where a Rust entropy monitor watches per-token uncertainty during local generation and routes to Sonnet only when the local model is genuinely stuck. (I know Anthropic has the advisor system, but this is different)

GitHub: https://github.com/Manojython/buddy-system

How it works

Gemma 3 4B generates locally on Apple Silicon via MLX. A Rust EntropyMonitor (compiled as a PyO3 extension) computes Shannon entropy over the full token vocabulary on every generated token:

I built a Rust entropy monitor to route LLM inference — here's what the benchmark showed

Related reading

We measured a week of inference. Routing by task difficulty cuts our cost per…

Running LLMs Locally on Consumer Hardware — Part 1: The Stack and First…

What I learned trying to benchmark local LLMs honestly

LiteLLM Is Moving to Rust. Here's What the Benchmarks Look Like.

LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC

How I Fixed LLM Hallucinations on a 512MB Server with Pure Math