I have been running a long series of experiments on whether small models can be made to behave usefully, and one obvious thing to borrow is whatever TinyStories did. That paper trained very small transformers to write coherent children's stories, and the detail everyone remembers is the vocabulary: roughly 1,500 words.
An earlier experiment of mine had already varied the domain at a fixed vocabulary, restricting a corpus to a single arXiv category, and found a real win: +3.7 points of top-1, perplexity down 1.8x, and 17x less degeneration into repetition. So the natural follow-up is the other axis. Hold the domain and vary the vocabulary. If the vocabulary is the lever, capping it should help.
It does not help; it hurts, by 4 points, and it wrecks the generated text. The research repo is not public, so the figures below come from the experiment's own harness and its recorded results.json rather than from anything you can clone.
The fairness problem, first
A smaller vocabulary makes next-token prediction trivially easier, because everything outside the cap becomes <unk> and <unk> becomes a frequent, easy target. Any arm with a smaller vocabulary will look better on a naive score, and it will look better for a reason that has nothing to do with capability.
