What Changed
Traditional benchmarks for visual document understanding, such as DocVQA and ChartQA, often combine multiple factors like document length, layout complexity, and question difficulty, making it challenging to pinpoint the exact causes of Vision Language Model (VLM) failures. To address this, a new benchmark called SynthDocBench has been introduced. This fully synthetic benchmark for long-context visual document understanding systematically controls these factors, enabling a more granular analysis of VLM behavior.
SynthDocBench is constructed using a combinatorial design, where each factor is independently varied across generated documents. This approach allows for controlled experimentation and analysis of how VLMs perform under specific conditions. The documents are generated end-to-end using an LLM pipeline, encompassing six distinct layout archetypes. A 40 percent random override is incorporated during generation to prevent models from exploiting spurious correlations that might arise from predictable synthetic data. Crucially, SynthDocBench extends to long-context documents with significantly greater length and structural diversity compared to existing benchmarks, pushing the boundaries of current VLM evaluation.
