When your benchmark is wrong and your model is right

When your benchmark is wrong and your model is right

We fine-tuned a 30-billion-parameter model on Barbados newspapers. Then we had to write three different benchmarks to figure out whether it actually got better. It did — but not in the way any single benchmark could show on its own, and one of them made it look worse before it looked better.

This is about the benchmarks, not the model. The model is Qwen3-Omni-30B-A3B-Instruct with a LoRA adapter trained on a cleaned Barbados newspaper archive. We're building Pulse, a live public-signal intelligence system for Barbados — radio, newspapers, TikTok, and government feeds turned into a queryable knowledge graph — as part of Future Caribbean, a 21-day agentic AI buildathon. Pulse is still in active development. The fine-tuned model is one of its domain-knowledge components, and it's also published openly for anyone working on Caribbean-domain text.

Three benchmarks, three answers

The model went through two versions. V3 stopped halfway through its training schedule (step 1000 of 1624, when the 12-hour GPU budget ran out). V4 resumed and completed the full schedule to step 1624. We scored both against the unmodified base model on three different eval tracks.