How to Fine-Tune DINOv3 on Custom Dataset with Self-Supervised Learning Locally

[MaybeRichard]

Hi team, I'm new to DINO and want to fine-tune DINOv3 using self-supervised learning on my custom dataset. Hardware: 8 NVIDIA 4090 GPU. Goal: Continue training from pre-trained checkpoints. Use the model for feature extraction and downstream tasks on medical images.
Dataset details:

• ~200k medical 2D images, mostly grayscale (plan to convert to RGB by duplicating channels).
• Various sizes/resolutions, formats (JPEG/PNG).

Questions:

• Recommended model size (e.g., ViT-B/16) for ~200k images?
• Training command example starting from checkpoints in repo?

[baldassarreFe]

Hello and thanks for the interest.

Before trying to fine-tune the backbone, can you try extracting the features from the frozen backbone and evaluating the downstream performance? Just in case it already works well.

If you want to fine-tune the model you can follow the steps outlined in the Training section of the readme: https://github.com/facebookresearch/dinov3?tab=readme-ov-file#training. A few notes:

• Start from the the ViT-L checkpoint, make sure it's loaded before starting to train.
• Use the ViT-L configuration vitl_im1k_lin834.yaml.
• Adapt the resolution to something that makes sense for your data and tasks. E.g. if it's segmentation you should start training at 512 and possibly increase to 768.
• Original img format (jpg/png) doesn't matter, as long as PIL can load them.

[halecakir]

Hi! I have a similar question.

I’d like to continue pretraining from one of the shared checkpoints (e.g., dinov3_vits16_pretrain_lvd1689m-*.pth).

1. Which argument should I use to resume?
   a) Should I use student.resume_from_teacher_chkpt for this, or is that flag only for the high-res adaptation stage?
   b) If it’s only for High-resolution adaptation, then to continue pretraining I assume I should resume from a full training checkpoint (with optimizer/EMA). However, the public files seem to be backbone-only.
   c) When I try resume_from_teacher_chkpt with the public file, loading looks like it expects a top-level "teacher" key (e.g., ckpt["teacher"] in init_fsdp_model_from_checkpoint), which those public weights don’t have.

2. What’s the intended usage of student.pretrained_weights?
   a) Should this be used to initialize a new pretraining run from the public backbone?
   b) Is student.pretrained_weights considered deprecated?

3. Recommended strategy for domain adaptation via continued training (no optimizer state available, etc)
   Since the public files don’t include optimizer state, last LR, last iter, etc., what config tweaks do you recommend to safely continue training on a new domain (medical images)?
   For example: start with a lower LR (how much lower than your default?) etc.

Thanks!

[MaybeRichard]

Thanks a lot!. By the way, could you please clarify what range of loss values is typically expected for DINOv3 training? In other words, around what loss level would you consider the model to be training well, assuming the dataset and setup are reasonable?

[alexisdrakopoulos]

@baldassarreFe thank you so much for allowing SSL fine-tuning this time! I can't wait to try it.

[alexisdrakopoulos]

@MaybeRichard have you managed to get this up and running? I am looking to SSL fine tune the base model on 3 million images soon but I'm very confused by how to start from the checkpoint and the configs I need.

[MaybeRichard]

@alexisdrakopoulos Yes, but there still have some bugs and I'm trying to fix, maybe be we can discuss furthur about how to make this work (DM me with email or others)

[alexisdrakopoulos]

@MaybeRichard I've emailed you. I won't be able to test this until later this week. But I definitely want to try fine tune using SSL for my out of domain dataset.

I still need to do some work setting up benchmarks though!

[sehunfromdaegu]

In my experience, training from scratch with domain-specific dataset performed better than fine-tuning the (general) pretrained model.

[alexisdrakopoulos]

@sehunfromdaegu but in this case the training procedure is quite slow / difficult due to the 7b model size. You can't just train the distilled models from scratch on a small dataset right? Then you might as well use dinov2? Or am I wrong

[sehunfromdaegu]

@sehunfromdaegu but in this case the training procedure is quite slow / difficult due to the 7b model size. You can't just train the distilled models from scratch on a small dataset right? Then you might as well use dinov2? Or am I wrong

You are right; it is quite slow and difficult to train a large model. We typically don't have very large domain-specific datasets or enough GPU power.

My experience is that pretraining a smaller model like ViT-B or ViT-L on domain-specific data outperforms a larger, general-purpose pretrained model like ViT-H. Honestly, I haven't tried fine-tuning ViT-G or ViT-7B, but I would guess that a smaller, domain-specific model would still work better. But you have to carefully choose the pretraining method, as SOTA (like DINOv2/DINOv3) pretraining approach may not work best for your specific domain.

[alexisdrakopoulos]

@sehunfromdaegu I have up to 6 or 7 million image / text pairs.

I tried fine-tuning SigLIP2 with 3 million but I had a lot of trouble with the performance stalling after 1 or 2 epochs, even with extremely low LRs.

Would you recommend trying to train a model from scratch? I only have access to 1xH200 for now and only for up to around 1 day which is frustrating.

[alexisdrakopoulos]

@sehunfromdaegu sorry just to elaborate.

When you talk about training a model from scratch what procedure are you using? The dinov2? Dinov3? Clip? Siglip? There are so many ways to train these ViT backends I can't decide what will work best.

The data augmentation procedures from dinov2 are perfect for my use cases. Maybe I need to look into it's training procedure along with CLIP models.

[sehunfromdaegu]

@sehunfromdaegu sorry just to elaborate.

When you talk about training a model from scratch what procedure are you using? The dinov2? Dinov3? Clip? Siglip? There are so many ways to train these ViT backends I can't decide what will work best.

The data augmentation procedures from dinov2 are perfect for my use cases. Maybe I need to look into its training procedure along with CLIP models.

A 1-day limit can be quite frustrating; perhaps implementing a way to resume from the latest checkpoint would be helpful.

I haven't actually tried pretraining with the DINOv2 framework. I believe its impressive performance comes partially from its specific data curation (and the size as well), which may not be suitable for my dataset.

Surprisingly, I found that a less popular pretraining method (VICRegL) worked better than other fancier approaches (e.g., Hiera) in a specific domain (but the number of pretraining samples were only about 1M). While hyperparameter choices may change the result, I now believe that success really depends on your dataset. Still, I suppose luck plays a pretty significant role as well.

[alexisdrakopoulos]

@sehunfromdaegu I'm trying to keep training cheap / lightweight for my use cases.

Maybe I should just try training dinov2 or v3 base models from scratch but it's very difficult. I saw dinov2 was trained successfully on sub 1m datasets with x-ray images.

It's annoying because I have good high quality text labels but I have had terrible experience trying to finetune SigLIP2. Maybe I need to try training from scratch.