Your Mixture-of-Experts fine-tune looks clean in eval and degrades in production, and the degradation tracks concurrency rather than input difficulty. Nothing throws. Logits look normal. The usual suspects — sampling params, quantization, prompt drift — all check out. The culprit is often the MoE capacity factor: a fixed-size buffer per expert that silently discards tokens when the router sends too many of them to the same place. A dropped token doesn't error. It just skips the feed-forward block at that layer and continues as if nothing happened.
Key takeaways
Every MoE layer allocates each expert a fixed buffer: capacity = capacity_factor × tokens × top_k / num_experts. Tokens that arrive after the buffer fills are dropped — they bypass the expert MLP and pass through on the residual stream alone.
Routers self-reinforce ("rich get richer"), so load imbalance is the default state, not an anomaly. Auxiliary load-balancing loss and router z-loss exist to fight it.
Drop rate depends on the token distribution in the current batch, which is why the failure appears under production traffic mixes and not in single-sequence eval.
