Skip to content

structured_outputs: support oneOf/anyOf branches and $defs/$ref schemas - #1103

Open
vincenth777 wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
vincenth777:fix/complex-schema-support
Open

vincenth777 wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
vincenth777:fix/complex-schema-support

Conversation

@vincenth777

Copy link
Copy Markdown

Two fixes to resources_servers/structured_outputs/app.py so the verifier handles complex JSON Schema features that it currently mishandles:

  1. strictify_schema: recurse into list items so object schemas nested inside oneOf / anyOf / allOf branches get required and additionalProperties: false applied. Without this, extra or missing fields inside a composite branch are not caught by validation.
  2. coerce_xml_types: resolve $ref pointers against the top-level schema's $defs before coercion, via a new root_schema parameter. Without this, $ref-based fields with non-string types (integer, number, boolean) are not coerced and fail validation.

Both changes are backward compatible: strictify_schema still handles plain dict schemas identically, and coerce_xml_types gains a root_schema=None default so existing call sites are unaffected.

Test plan

  • test_verify_json_with_oneof_strictifies_branches: valid branch passes; extra/missing field inside a oneOf branch rejected
  • test_verify_xml_with_defs_refs: integer under $ref coerces correctly; non-numeric content rejected
  • All 5 existing test_app.py tests still pass (JSON, YAML, XML, TOML, CSV)

Context

Surfaced while building structured-output tasks with complex schemas, context discussed with JK

  - strictify_schema: recurse into list items so object schemas nested inside
    oneOf/anyOf/allOf branches get required + additionalProperties: false.
  - coerce_xml_types: resolve $ref pointers against the top-level schema's
    $defs via a new root_schema parameter (defaults to None for backward compat).

Signed-off-by: Vince Huang <vincehuang@mercor.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

- strictify_schema: skip allOf branches (conjunctive merge would reject
  fields contributed by sibling branches)
- coerce_xml_types: support nullable union types like ["integer","null"]
  (parity with CSV path); make root_schema required; use removeprefix
- add tests: anyOf strictification, allOf composition regression guard,
  nullable XML coercion; extend oneOf test with second-branch case
- add data/example_complex_schema.jsonl (5 toy rows) and an end-to-end
  smoke test that round-trips every row with hand-crafted gold outputs

Signed-off-by: Vince Huang <vincehuang@mercor.com>
@vincenth777

Copy link
Copy Markdown
Author

allOf safety strictify_schema now short-circuits on the allOf key. Branches merge conjunctively, so adding additionalProperties: False to one branch would reject fields contributed by siblings. New regression-guard test test_verify_json_with_allof_does_not_overstrictify_branches exercises a valid composition spanning both branches.

anyOf test coverage: added test_verify_json_with_anyof_strictifies_branches covering valid match (first branch + second branch), missing-required rejection, extra-field rejection, and no-branch-match rejection.

Nullable XML coercion: coerce_xml_types now handles list-valued type like ["integer", "null"] and empty tags map to None, populated values coerce against the first non-null member. Mirrors _coerce_csv_scalar. New test test_verify_xml_with_nullable_union_types covers populated, empty, and invalid-content cases.

Toy examples: added data/example_complex_schema.jsonl with five rows (oneOf, anyOf, allOf, \$defs/\$ref XML, nullable XML). The new test_example_complex_schema_jsonl_round_trips loads the file and runs every row end-to-end through evaluate_structured_output_response with hand-crafted gold outputs, asserting reward=1.0 per row.

Other small cleanups inside the new code:

  1. \$ref resolution uses removeprefix("#/") instead of lstrip (correctness — lstrip strips a charset, not a prefix)
  2. root_schema parameter on coerce_xml_types is now required, eliminating a silent ref-resolution no-op for external callers

Verified empirically: swapped app.py to upstream main, ran the test suite, and confirmed the four capability tests + the smoke test all fail on main with the exact failure modes you flagged. Restored, all 11 pass.

@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Jul 17, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 20, 2026
@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test 9bd5897

@cmunley1

Copy link
Copy Markdown
Contributor

/review

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Oct 30, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working community-request Issue reported or requested by someone from the community needs-review PR is ready for code review and waiting on a reviewer sla:triage-overdue Review assignment is over the one-business-day SLA waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants