Usage and License Notices: This project is based on LLaVA-OneVision codebase. Please refer to their usage and license notices for more information.

Models & Scripts

Installation

1. Clone this repository and navigate to the LLaVA folder:

git clone https://github.com/ngailapdi/SplatTalk-LLaVA-Inference
cd SplatTalk-LLaVA-Inference

2. Install the inference package:

conda env create -f environment.yml
conda activate llava
pip install --upgrade pip  # Enable PEP 660 support.
pip install -e ".[train]"

Finetuning

1. On SLURM-based clusters

bash cluster.sh

2. Local

bash scripts/train/finetune_ov.sh

Inference

python scripts/decode_dataset.py

Evaluate

python scripts/scanqa_evaluator.py

Extract LLaVA-OV features for training SplatTalk

python scripts/extract_dataset_feat.py

Acknowledgement

• This codebase is built upon LLaVA-OneVision. We thank the authors for their great work!