Skip to content

[Data][LLM] Support OpenAI's nested image_url schema in PrepareImageStage #56584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kouroshHakha merged 4 commits into from
Oct 2, 2025
Merged

[Data][LLM] Support OpenAI's nested image_url schema in PrepareImageStage #56584

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6d62f59
[Data][LLM] Support openai's nested image_url format in PrepareImageS…
GuyStone Sep 16, 2025
259b7e5
Merge branch 'master' into support-openai-nested
GuyStone Sep 24, 2025
0daec85
add notes + add tests + lint
nrghosh Oct 2, 2025
bab1515
Merge branch 'master' into support-openai-nested
nrghosh Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions python/ray/llm/_internal/batch/stages/prepare_image_stage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Prepare Image Stage"""

import asyncio
import base64
import importlib
Expand Down Expand Up @@ -311,7 +312,7 @@ def __init__(self, data_column: str, expected_input_keys: List[str]):
self.image_processor = ImageProcessor()

def extract_image_info(self, messages: List[Dict]) -> List[_ImageType]:
"""Extract vision information such as image and video from chat messages.
"""Extract image information from chat messages.

Args:
messages: List of chat messages.
Expand All @@ -336,7 +337,18 @@ def extract_image_info(self, messages: List[Dict]) -> List[_ImageType]:
for content_item in content:
if content_item["type"] not in ("image", "image_url"):
continue
image = content_item[content_item["type"]]

image_data = content_item[content_item["type"]]

if content_item["type"] == "image_url" and isinstance(image_data, dict):
# OpenAI nested format: {"image_url": {"url": "..."}}
image = image_data.get("url")
if image is None:
raise ValueError("image_url dict must contain 'url' key")
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This is good validation to ensure the image_url dictionary is well-formed. To make this more robust, it would be beneficial to add a unit test case that verifies this ValueError is raised when the url key is missing. This would improve test coverage for error handling paths.

else:
# Simple format: {"image": "..."} or {"image_url": "..."}
image = image_data

if not isinstance(image, str) and not isinstance(
image, self.Image.Image
):
Expand Down
18 changes: 18 additions & 0 deletions python/ray/llm/tests/batch/cpu/stages/test_prepare_image_stage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,24 @@ async def test_prepare_image_udf_invalid_image_type(mock_image_processor):
["https://example.com/image.jpg"],
"image_url_format_no_system_prompt",
),
# Test OpenAI nested format without system prompt
# https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml#L1937-L1940
(
[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": "https://example.com/image.jpg"},
},
{"type": "text", "text": "Describe this image"},
],
}
],
["https://example.com/image.jpg"],
"openai_image_url_format_no_system_prompt",
),
],
ids=lambda x: x if isinstance(x, str) else None,
)
Expand Down