A moderation system is not a classifier. It is a pipeline with a cheap filter, an expensive opinion, a human queue, an appeals path and a log you can put in front of a regulator. The classifier is the part that takes an afternoon; the other four are what makes the thing operable and defensible.

Why two stages

The arithmetic decides this, and it is worth doing before writing any code.

Traffic: 1,000,000 items per day, of which ~0.5% violate policy.

Single stage, one capable model on everything: