Train separately, merge together: Modular post-training with mixture-of-experts | Ai2

BAR is a recipe for post-training language models one capability at a time—train domain experts independently, merge them into a single mixture-of-experts model, and upgrade any expert without impacting the others.

Raccontata daallenai.org