Benchmark/comparison against MiniGPT-4?

[152334H]

Or just thoughts about which architecture should be expected to perform better with the same training data

[ChunyuanLI]

It is great to see the community's recognition and excitement about this direction; Both pieces of work are taken independently during the same period.

IMHO, LLaVA is unique at three aspects, see below.

1. More rigorous results: LLaVA has rigorous quantitative results, including the level of similarity with Visual Chat and GPT-4, the SoTA accuracy on Science QA, and ablation studies on data iteration and model design. Mini GPT-4, on the other hand, lacks quantitative results.
2. Quality of Chat Demo: LLaVA can reproduce results for visual reasoning examples in GPT-4 paper and has strong OCR capabilities. These features are impressive and unique, making it possibly the closest demo to Multimodal GPT-4. Check results: https://llava-vl.github.io
3. Lastly, it should be clarified that the focus of this line of work is data-centric, not model-centric. As the differences in models are diminishing, data quality has a greater impact on results. We released our multi-modal instruction following data, to replicate Multimodal GPT-4. The high quality data is all you need (compare which, the architecture is secondary).

[ChunyuanLI]

Comparison of LLaVA and mini-GPT4 on "Extreme Ironing" example from the OpenAI GPT-4 technique report.

LLaVA

---

mini-GPT4

Run 1:

Run 2: