Short answer: move marketplace review summarization, tagging, and extraction to batch LLM jobs when no customer is waiting, but keep realtime calls for interactive work and attribute every job to a tenant before it enters the queue.
This is a deadline decision before it is a vendor decision. A nightly policy scan can wait; a seller asking why a listing was rejected cannot. Batch processing removes peak-time synchronous handling from the first case and gives the team a status-and-results workflow for backfills. It does not make latency disappear.
The other constraint is accounting. A marketplace that pools every review into one opaque job may lower operational friction while making chargeback, abuse investigation, and budget alerts much harder. The useful unit is therefore a tenant-scoped batch with an internal ledger entry, not merely a large file of prompts.
Treat every finding as governed evidence
Create the ledger record before dispatch. It should connect an immutable internal job ID to the tenant, workload kind, input count, model choice, submission time, deadline, and estimated token total. Keep the provider job ID as a later mapping rather than using it as your primary key. That leaves audit history and cost attribution intact if the team changes providers.
