[ Request ] Multimodal model generation

[WoutDeRijck]

Now it is only possible to pass text to enforced model. Some tasks require multimodal models, like processing images. It would be awesome if it would be possible to enforce a multimodal model to use a particular JSON schema.

[ayylemao]

If you use Hugginface's transformers library it is possible. Just add the prefix function to the generate method and it will work:

            generated_ids = model.generate(**inputs, 
                                           max_new_tokens=256, 
                                           do_sample=False, 
                                           prefix_allowed_tokens_fn=prefix_func)```

[WoutDeRijck]

I am currently working with miniCPM:

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='flash_attention_2', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

parser = JsonSchemaParser(InvoiceFormat.schema())
prefix_function = build_transformers_prefix_allowed_tokens_fn(tokenizer, parser)

When I pass this prefix function to the model.generate I get these kinds of errors:

file /opt/conda/lib/python3.10/site-packages/transformers/generation/logits_process.py:1337, in PrefixConstrainedLogitsProcessor.__call__(self, input_ids, scores)
   1334 @add_start_docstrings(LOGITS_PROCESSOR_INPUTS_DOCSTRING)
   1335 def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
   1336     mask = torch.full_like(scores, -math.inf)
-> 1337     for batch_id, beam_sent in enumerate(input_ids.view(-1, self._num_beams, input_ids.shape[-1])):
   1338         for beam_id, sent in enumerate(beam_sent):
   1339             prefix_allowed_tokens = self._prefix_allowed_tokens_fn(batch_id, sent)

RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 1, 0] because the unspecified dimension size -1 can be any value and is ambiguous

[WoutDeRijck]

Here is a small notebook I made to show where it goes wrong.

lmformatenforcer.ipynb.zip

[noamgat]

There is a new sample that shows multimodal extraction working:
https://github.com/noamgat/lm-format-enforcer/blob/main/samples/colab_llama32_vision_enforcer.ipynb