Repository navigation
[WIP] Add Ovis2.5 - #40321
[WIP] Add Ovis2.5 #40321thisisiron wants to merge 4 commits into
Conversation
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto |
|
Reviewing, sorry for delay |
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
Not sure if the PR is ready for review. Just left a few general comments
| The is an updated version of the [Ovis](https://arxiv.org/abs/2405.20797) model developed by the AIDC-AI team at Alibaba International Digital Commerce Group. | ||
|
|
||
| [Ovis2.5](https://arxiv.org/abs/2508.11737) is a multimodal vision-language model, available in 2B and 9B parameter versions. It introduces a native-resolution vision transformer (NaViT) that processes images at their original resolutions, preserving both fine details and global layout—crucial for visually dense inputs like complex charts and documents. The model also features a novel reflective reasoning capability, including self-checking and revision, accessible via an optional thinking mode at inference time. | ||
|
|
There was a problem hiding this comment.
Lte's add an abstract in italics to follow general model doc guidelines
| from transformers import AutoModelForVision2Seq, AutoTokenizer, AutoProcessor | ||
|
|
||
| model = AutoModelForVision2Seq.from_pretrained( | ||
| "", |
| from transformers.image_utils import load_images, load_video | ||
| from transformers import AutoModelForVision2Seq, AutoTokenizer, AutoProcessor | ||
|
|
||
| model = AutoModelForVision2Seq.from_pretrained( |
There was a problem hiding this comment.
AutoModelForVision2Seq is deprecated, we need to ise AutoModelForImageTextToText
| url = "http://images.cocodataset.org/val2014/COCO_val2014_000000537955.jpg" | ||
| image = Image.open(requests.get(url, stream=True).raw) | ||
| messages = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | ||
| print(messages) | ||
|
|
||
| inputs = processor( | ||
| images=[image], | ||
| text=messages, | ||
| return_tensors="pt", | ||
| ) | ||
| inputs = inputs.to("cuda:0") | ||
| inputs['pixel_values'] = inputs['pixel_values'].to(torch.bfloat16) | ||
|
|
There was a problem hiding this comment.
can do shorter with
processor.apply_chat_template(messages, tokenize=True, return_dict=True, return_tensors='pt')
See llava docs for an example
| print(output_text) | ||
| ``` | ||
|
|
||
| <!-- TODO: Convert Ovis2 to Ovis2.5 --> |
| # Apply patch embeddings to already patchified pixel values | ||
| target_dtype = self.patch_embedding.weight.dtype | ||
| if isinstance(self.patch_embedding, nn.Linear): | ||
| patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) | ||
| elif isinstance(self.patch_embedding, nn.Conv2d): | ||
| pixel_values = pixel_values.view( | ||
| -1, self.config.num_channels * self.config.temporal_patch_size, self.patch_size, self.patch_size | ||
| ) | ||
| patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) | ||
| patch_embeds = patch_embeds.reshape(-1, self.embed_dim) |
There was a problem hiding this comment.
continuing on above comment, this looks weird if we are checking on module type. In case we need both position embedding types, i thing it's better to have separate classes for each and just call it
| # pos_embed = self.pos_embed.reshape(1, ori_h, ori_w, -1).permute(0, 3, 1, 2) | ||
| cnt = 0 | ||
| for t, h, w in grid_thws: | ||
| thw = t * h * w | ||
| pe = F.interpolate(positional_embeddings, size=(h, w), mode="bicubic", align_corners=False) | ||
| pe = pe.permute(0, 2, 3, 1).reshape(1, h * w, -1) | ||
| pe = pe[0].repeat(t, 1) | ||
| pe = pe.reshape( | ||
| t, h // self.hidden_stride, self.hidden_stride, w // self.hidden_stride, self.hidden_stride, -1 | ||
| ) | ||
| pe = pe.permute(0, 1, 3, 2, 4, 5).reshape(thw, -1) | ||
| pos_embed_new[cnt : cnt + thw] = pe | ||
| cnt += thw |
There was a problem hiding this comment.
could be in self.interpolate_position_embeddings(grid_twh) to be aligned with other ViT models
| None if config.fullatt_block_indexes is None else [int(i) for i in config.fullatt_block_indexes.split("|")] | ||
| ) | ||
|
|
||
| def rot_pos_emb(self, grid_thw): |
There was a problem hiding this comment.
looks like the rope from Qwen2.5-VL, maybe we should use modular with Qwen2.5-Vision then?
| r""" | ||
| Args: | ||
| inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): | ||
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. | ||
| This is useful if you want more control over how to convert `input_ids` indices into associated vectors | ||
| than the model's internal embedding lookup matrix. | ||
| attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): | ||
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | ||
| - 1 for tokens that are **not masked**, | ||
| - 0 for tokens that are **masked**. | ||
| [What are attention masks?](../glossary#attention-mask) | ||
| output_attentions (`bool`, *optional*): | ||
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under | ||
| returned tensors for more detail. | ||
| output_hidden_states (`bool`, *optional*): | ||
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors | ||
| for more detail. | ||
| return_dict (`bool`, *optional*): | ||
| Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. |
There was a problem hiding this comment.
let's use @autodocstring instead to auto-generate docs
| if self.gradient_checkpointing and self.training: | ||
| hidden_states = self._gradient_checkpointing_func( | ||
| block.__call__, hidden_states, cu_seqlens_tmp, position_embeddings | ||
| ) | ||
| else: | ||
| hidden_states = block(hidden_states, cu_seqlens_tmp, position_embeddings) |
There was a problem hiding this comment.
GC can be handled better by using GradientCheckpointingLayer :)
|
@zucchini-nlp, I’m still working on adding the model. Sorry for the confusion. I’ll ping you once it’s ready for review. Thank you for taking the time to review. |
|
Oke, i will unsubscribe myself then :) |
|
@thisisiron hey, is this PR still active? Ovis2.5 is getting community interest, so lmk if you need help with finalizing the PR |
|
@zucchini-nlp, I’ve been quite busy recently and haven’t been able to make proper progress on this. Apologies for the delay. Please feel free to close the PR so that someone else can continue the work. |
|
No probs :) In case you want to take over and finish the PR, cc @mahimairaja. The next step would be to address comments from last review |
What does this PR do?
This PR adds support for Ovis2.5, a multimodal vision-language model designed for native-resolution visual perception and reflective reasoning.
Ovis2.5 is released in two variants:
This is still a draft PR. Further integration and testing to follow.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.