2023 VisualInstructionTuning
- (Liu, Li et al., 2023) ⇒ Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. (2023). “Visual Instruction Tuning.” In: arXiv preprint arXiv:2304.08485. doi:10.48550/arXiv.2304.08485
Subject Headings: Multimodal Language-Image Model, Instruction-Tuned LLM, LLaVA Model.
Notes
Cited By
Quotes
Abstract
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for [[general-purpose visual and language understanding. Our early experiment]]s show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images / instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2023 VisualInstructionTuning | Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee | Visual Instruction Tuning | 10.48550/arXiv.2304.08485 | 2023 |