[Bug] Parameter insertion_point not working as intended for shared_prefix multimodal config

[rahulgurnani]

Unexpected behavior
Setting insertion point as 0.5 doesn't lead to image being inserted in the middle of the requests based on the description of the flag in the code.

Config to reproduce

api:
  type: completion
  streaming: false
data:
  type: shared_prefix
  shared_prefix:
    # Using both names for clarity; num_unique_system_prompts takes precedence if both exist.
    num_groups: 20
    num_prompts_per_group: 1
    system_prompt_len: 0
    question_len: 200
    output_len: 50
    enable_multi_turn_chat: false
    multimodal:
      image:
        count:
          type: fixed
          min: 2
          max: 2
          mean: 2.0
        insertion_point: 0.5
        resolutions:
        - resolution: 720p
          weight: 1.0
load:
  type: constant
  stages:
  - rate: 1.0
    duration: 10
server:
  type: vllm
  base_url: http://IP:80

all the requests start with:

{"model": "mock_model", "messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"..."}... 

I was expecting them to start with text.

cc @Bslabe123 this seems like a bug.

cc @capri-xiyue

[Bslabe123]

You've set data.shared_prefix.multimodal which is causing images to be woven into prefixes of length system_prompt_len=0, you're likely looking for data.multimodal which configures how images are being woven into the questions

[rahulgurnani]

You've set data.shared_prefix.multimodal which is causing images to be woven into prefixes of length system_prompt_len=0, you're likely looking for data.multimodal which configures how images are being woven into the questions

Do you have an example config to generate such data?

[Bslabe123]

See attached config:

  api:
    type: completion
    streaming: false
  data:
    type: shared_prefix
    shared_prefix:
      num_groups: 20
      num_prompts_per_group: 1
      system_prompt_len: 0
      question_len: 200
      output_len: 50
      enable_multi_turn_chat: false
    # Payload-side multimodal: images are woven into the *question* text,
    # not the shared prefix. insertion_point is a fraction of the question length.
    multimodal:
      image:
        count:
          type: fixed
          min: 2
          max: 2
          mean: 2.0
        insertion_point: 0.5   # ~char 100 of the 200-char question
        resolutions:
        - resolution: 720p
          weight: 1.0
  load:
    type: constant
    stages:
    - rate: 1.0
      duration: 10
  server:
    type: vllm
    base_url: http://IP:80