AttributeError: BartTokenizerFast has no attribute image_token. Did you mean: 'mask_token'?

[conceptofmind]

System Info

Ubuntu

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
import requests
from PIL import Image
from transformers import AutoProcessor, Florence2ForConditionalGeneration


model = Florence2ForConditionalGeneration.from_pretrained(
    "microsoft/Florence-2-large",
    dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large")

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

task_prompt = "<OD>"
inputs = processor(text=task_prompt, images=image, return_tensors="pt").to(model.device, torch.bfloat16)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

image_size = image.size
parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=image_size)

print(parsed_answer)

Expected behavior

    raise AttributeError(f"{self.__class__.__name__} has no attribute {key}")
AttributeError: BartTokenizerFast has no attribute image_token. Did you mean: 'mask_token'?

[conceptofmind]

transformers>=4.53.3

[Rocketknight1]

Seeing lots of missing weights in that checkpoint too - cc @zucchini-nlp if you can figure this one out!

[zucchini-nlp]

Oh, for the non-remote code loading we need to use converted weights which are in https://huggingface.co/florence-community. Let me check if the docs are all updated, but it should have been already

[conceptofmind]

How should something such as this be converted, then? https://huggingface.co/MiaoshouAI/Florence-2-large-PromptGen-v2.0

I imagine these changes break compatibility with numerous previously trained models. Proper conversion should be added to the documentation.

[zucchini-nlp]

You have the conversion scripts available in model folders usually, for Florence it is in https://github.com/huggingface/transformers/blob/main/src/transformers/models/florence2/convert_florence2_original_pytorch_to_hf.py

It was used to convert the microsoft weights, so if the finetuned models has the same format as the official checkpoint you can use the above script