I set out to deploy a fine-tuned LLM fully offline, on a mid-range Android phone, answering medical questions in Yoruba, Hausa, Igbo, and Nigerian Pidgin. No internet connection required, because that's the reality for a lot of the people this was meant to help.
The model worked. Getting there broke three times, in three completely different ways, and left me with one problem I still haven't solved.
The setup
I fine-tuned unsloth/Llama-3.2-3B-Instruct, Unsloth's 4-bit-optimized derivative of Meta's Llama 3.2 3B, in two stages: QLoRA supervised fine-tuning on a curated dataset of 3,917 medical question-answer pairs across the four languages, followed by direct preference optimization (DPO) to sharpen response quality. SFT converged to a loss of 1.099. DPO landed a reward margin of 18.40.
Then I merged to 16-bit and tried to convert to GGUF, the format llama.cpp needs to run the model on-device. That's where things started breaking.
