[Bugfix] Dictionary MM embeddings for online chat

[DarkLight1337]

Purpose

• Handle multiple dictionary inputs in MultiModalItemTracker.all_mm_data and its async version.
• Update InputProcessor._validate_single_prompt to handle dictionary inputs correctly.

FIX #29778

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds support for multiple dictionary-based input embeddings, which is a great enhancement. The core logic changes in chat_utils.py and input_processor.py seem to correctly handle the main use case of multiple dictionary embeddings. The new tests for this functionality are also valuable.

However, I've identified two critical issues. First, the refactoring seems to have broken the existing functionality for handling cached embeddings (where image_embeds is null but a uuid is provided), as the new _get_embeds_data function does not handle None values. Second, the new test for empty dictionary embeddings ({}) is internally inconsistent and reveals a bug that will cause a validation failure. An empty dictionary is treated as one item, creating a placeholder, but its data length is considered zero, leading to a mismatch with the UUID count. Please see my detailed comments for suggestions on how to address these issues.

[mergify]

Hi @DarkLight1337, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a bugfix to correctly handle dictionary-based multimodal embeddings, particularly for online chat scenarios. The changes involve updating MultiModalItemTracker and InputProcessor to manage multiple dictionary inputs. New tests have been added to cover these cases. My review focuses on the robustness of the new logic. I've identified a potential KeyError in _get_embeds_data if input dictionaries have inconsistent keys and have suggested a validation to prevent this.

[DarkLight1337]

Test currently fails, will resume work tomorrow

[mergify]

Hi @DarkLight1337, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[DarkLight1337]

Tests should pass now