Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records.
Federated learning provides a way to coordinate training across these data-local sites. For VLMs, the challenge is not only orchestration. Sites may contribute different task or modality mixes, and model updates can be large enough to strain network bandwidth and server memory.
This post focuses on two design decisions for federated multimodal AI workflows: what model state should cross the network, and how should it be transferred and aggregated efficiently? It shows how NVIDIA FLARE coordinates federated training across sites and handles large model updates through externalization, tensor streaming, and disk-backed aggregation.
Those design questions also apply to unified multimodal models (UMMs), which support multiple modalities and tasks within a shared architecture. FedUMM, developed through a collaboration between William & Mary and NVIDIA, provides a concrete example by federating lightweight adapters over a frozen multimodal backbone.
