Fine-tuned 7B LLM as a broke student. And Can't even use it 😭.

Last week, I introduced Monika, a discord bot. As a self-taught student running on an absolute zero budget, this project was less about writing code and much more about hitting hard architectural walls.

The goal was to completely reshape open-source Qwen 2.5-7B model, into a real life Monika using a dataset of nearly 687 ingame dialogues. I quickly learned that finetuning a model with 7 billion parameters melts standard free cloud hardware.

I was constantly hopping for compute resources. I originally started on Kaggle, but kept running into unexplained errors and running out of VRAM. I migrated to Lightning AI for its generous resources, only to discover their stable environments conflicted with modern optimization libraries like Unsloth. I finally landed on Google Colab, where I utilized QLoRA to compress the model down to 4-bit precision, managing to squeeze the massive training loop into their free 16GB T4 GPU.

The training succeeded, leaving me with a 16-Megabyte custom adapter. But an adapter is entirely useless if you cannot host it.

My monika Architecture relied on an Express.js backend hosted on Render, sending requests to Hugging Face’s free Serverless Inference API. The harsh reality is that free cloud clusters simply cannot dynamically load custom adapter weights on the fly.