Dragonfly: A large vision-language model with multi-resolution zoom

We are excited to announce the launch of Dragonfly, an instruction-tuning Vision-language architecture, that enhances fine-grained visual understanding and reasoning about image regions. We are releasing the Dragonfly architecture, which uses multi-resolution zoom-and-select to enhance multi-modal reasoning while being context-efficient. We are also launching two new open-source models Llama-3-8b-Dragonfly-v1 a general-domain model trained on 5.5 million image-instruction pairs and Llama-3-8b-Dragonfly-Med-v1 finetuned on additional 1.4 biomedical image-instruction data. Dragonfly demonstrates promising performance on vision-language benchmarks like commonsense visual QA and image captioning. Dragonfly-Med outperforms prior models, including Med-Gemini on multiple medical imaging tasks, showcasing its capabilities for high-resolution medical data.Figure 1: Generated example from Llama-3-8B-DragonflyQuestion: Summarize the visual contents of the image.‍Dragonfly Response: In the heart of a vibrant skatepark, a skateboarder is caught in a moment of pure exhilaration. The skateboarder, dressed in a black t-shirt adorned with a yellow graphic and black pants, is suspended in mid-air, performing an impressive trick on a concrete ramp. The skateboarder's arms are outstretched, adding balance to the daring stunt. The skatepark itself is a concrete playground, with the skateboarder's ramp being the main focus. In the background, palm trees sway gently, adding a touch of nature to the urban setting. A few spectators can be seen in the distance, their attention riveted on the airborne skateboarder. The image captures not just a moment, but a story of skill, courage, and the joy of skateboarding.Figure 2: Generated examples from Llama-3-8B-Dragonfly-MedQuestion: Write a detailed radiology note based on the chest X-ray.‍Gold Answer: AP upright and lateral views of the chest were provided. Left chest wall pacer pack is again seen with leads extending into the right heart. Abandoned pacing leads are also noted in the right chest wall extending into the right heart. The heart remains moderately enlarged. Lung volumes are low, with equivocal ground-glass opacity on the frontal view, which appears less conspicuous on the lateral view most likely attributable to underpenetrated technique. No gross evidence for pneumonia or pulmonary edema. No large effusions are seen. There is no pneumothorax. Bony structures are intact.‍Dragonfly Response: PA and lateral views of the chest provided. Left chest wall pacer device is again noted with leads extending into the region the right atrium and right ventricle. The heart remains mildly enlarged. There is mild hilar congestion without frank edema. No convincing signs of pneumonia. No large effusion or pneumothorax. Cardiomediastinal silhouette is stable. Bony structures are intact Dragonfly Model ArchitectureArxiv Paper with technical details and codebase for architecture implementationDragonfly is special because:Dragonfly employs two key strategies: multi-resolution visual encoding and zoom-in patch selection, which enables the model to focus more fine-grained details on image regions and provide better commonsense reasoning. Despite the model being optimized for capturing fine-grained image details, the model achieves good zero-shot performance at standard image-understanding benchmarks such as visual question-answering or natural-image-captioning.We demonstrate our model's understanding and reasoning capabilities on biomedical tasks, which often require a fine-grained understanding of high-resolution image regions. By finetuning our general-domain model on a biomedical instruction-tuning dataset with 1.4 million biomedical image-text pairs, Dragonfly-med, our biomedical version of the model, achieves state-of-the-art or competitive performance on multiple biomedical benchmarks, including visual question answering, image captioning, and radiology report generation.Figure 3: Overview of the Dragonfly architectureMulti-resolution Visual Encoding: Dragonfly processes images at multiple resolutions—low, medium, and high. Each image, depending on its resolution, is divided into sub-images that are encoded into visual tokens. These tokens are then projected into a language space, forming a concatenated sequence that feeds into the LLM. This method allows the model to handle larger images efficiently and improves the granularity of visual data processing.Zoom-in Patch Selection: To further refine the model's focus on important visual details, Dragonfly employs a selective approach for high-resolution images. It uses a novel zoom-in patch selection strategy to identify and retain only those high-resolution sub-images that provide the most significant visual information. This is achieved by comparing summary embeddings of medium and high-resolution sub-images to select the most semantically relevant patches. This targeted selection reduces redundancy and focuses on crucial content areas, thereby enhancing the overall model efficiency and fine-grained understanding on these areas.Together, these strategies enable Dragonfly to achieve a detailed and efficient visual understanding, making it particularly adept at processing complex image data in specific domains.Figures 4 and 5: Generated examples from Llama-3-8B-DragonflyFigure 4 Question: Summarize the visual content of the image first and then answer the questions. What process does this diagram portray and which solar body is portrayed in this diagram?Dragonfly Response: The diagram portrays the process of photosynthesis in plants, which is a fundamental biological process where plants convert light energy from the sun into chemical energy in the form of glucose. The solar body portrayed in this diagram is the sun, which is shown at the top left corner, providing the light energy that is essential for photosynthesis.Figure 5 Question: Summarize the image and why it is funny?Dragonfly Response: The image is a humorous take on the Mona Lisa painting, featuring a dog in the place of the human subject. The dog's expression, combined with the iconic pose and attire, creates a playful and amusing effect. The painting is executed in the same style as the original Mona Lisa, with similar brushwork and color palette, which adds to the humor by juxtaposing the serious art historical context with the light-hearted subject matter.Dragonfly Model EvaluationWe evaluate Dragonfly trained based on LLaMA-8B on five popular vision-language benchmarks that require strong commonsense reasoning and detailed image understanding, AI2D, ScienceQA, MMMU, MMVet, and POPE. AI2D and ScienceQA evaluate visual commonsense reasoning in the science domain. MMMU and MMVet focus on providing a comprehensive evaluation of the vision-language capabilities. POPE evaluates the object-level hallucination of the regional details. Dragonfly achieved competitive performance compared with other popular vision-language models, demonstrating the effectiveness of Dragonfly on commonsense reasoning and fine-grained understanding of image regions. The results are presented in the following table:

Raccontata datogether.ai