I wanted to build something with an LLM using my own hands. Not wire an API into a wrapper, but take a base model, train it, measure it, break it, and serve it over HTTP. Why? Because I wanted to get my hands dirty working with an LLM, making and breaking things.
Text to SQL fits exactly. You ask a question in plain English, the system writes SQL, and you can run the query to find out whether it was right. Correctness is whether the rows match the reference query's rows against the real database, which is a fact rather than a judgement. It also has a mature benchmark in Spider, 10,000 human written questions over 200 real SQLite databases, split so the test databases never appear in training.
So I built it. Qwen2.5-0.5B as the base, LoRA adapters, one g5.xlarge with an NVIDIA A10G, about a dollar an hour. Continued pretraining on SQL text, then supervised fine-tuning on question and query pairs, then reinforcement learning with GRPO on top. The dashboards looked great. Reward climbing to 1.0, loss dropping cleanly through every stage.
It scored 6.4%. Comically bad levels of accuracy.
I didn't find that out for a while, because there was no held out evaluation. Every number I had came from data the model had trained on. The untrained base model scored 17.4%, so three stages of training had made things worse than no training at all.
