Crash-Proof Batch LLM Processing: A SQLite Job Queue on a Free Server

Three thousand lines in, the process died. Not with a Python traceback — with a silent OOM kill. The input file was untouched. The 3,000 records already sent to the model were unmarked. Re-running meant paying for them twice.

Cron does not solve this. A for loop does not solve this. A queue does.

This article walks through a small log-normalization job — 10,000 unstructured error lines turned into structured JSON — using SQLite as the queue, Python as the worker, and MonkeyCode's free model access plus free server option as the infrastructure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Job