Skip to content

fix: tau2 knowledge deps + repoint tau2-bench pins to stable branches - #1954

Merged
wprazuch merged 5 commits into
mainfrom
edobrowolska/tau3
Jul 9, 2026
Merged

wprazuch merged 5 commits into
mainfrom
edobrowolska/tau3

Conversation

@e-dobrowolska

@e-dobrowolska e-dobrowolska commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Two fixes needed to run tau3 through NEL:

Verified end-to-end via NEL/EFB: 3x tau3-banking panda runs (8.04/9.69/7.01% vs 8.66% reference, within run-to-run noise) and qwen 8.87% (matches AA 8.7). Empty user responses occurred and recovered in-retry; zero hard-fails, zero ClientOSErrors.

Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cmunley1
cmunley1 previously approved these changes Jul 8, 2026
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@wprazuch wprazuch changed the title fix: minor fixes to launch tau3 evals with nel fix: tau2 knowledge deps + repoint tau2-bench pins to stable branches Jul 9, 2026
wprazuch added 3 commits July 9, 2026 20:00
… info counters

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…eeps secrets-detector quiet)

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

@wprazuch wprazuch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified end-to-end: 4x tau3-banking full runs via NEL/EFB against these pins (3x panda within noise of reference, qwen matching AA), tau2 tests green against bxyu/nemo_gym_stable.

@wprazuch
wprazuch merged commit f245781 into main Jul 9, 2026
9 checks passed
@wprazuch
wprazuch deleted the edobrowolska/tau3 branch July 9, 2026 18:10
laszkiewiczp added a commit that referenced this pull request Jul 10, 2026
The tau2-bench pin tracks a branch that moved schema (nl_assertions),
so the recorded request/response fixture on this base no longer matches
what the server produces; take the updated fixture from main's fix
commit f245781 without its runtime changes, which do not apply to this
base's tau2 layout.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
hkumar92 pushed a commit to hkumar92/Gym that referenced this pull request Jul 14, 2026
The tau2-bench pin tracks a branch that moved schema (nl_assertions),
so the recorded request/response fixture on this base no longer matches
what the server produces; take the updated fixture from main's fix
commit f245781 without its runtime changes, which do not apply to this
base's tau2 layout.

Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
OlegSudakov pushed a commit to OlegSudakov/Gym that referenced this pull request Jul 15, 2026
…NVIDIA-NeMo#1954)

Two fixes needed to run tau3 through NEL:

- Pin the tau2 `[knowledge]` extra deps (rank-bm25, openai) explicitly
in requirements.txt — the extra is not applied to a `tau2[knowledge] @
git+URL` direct reference, so bm25/bm25_grep environments miss rank-bm25
on install.
- Repoint the tau2-bench code pin (requirements.txt, source.py default,
README) to `bxyu/nemo_gym_stable` and the data ref mention to
`bxyu/nemo_gym_data` — the empty-user-message hard-fail handling (retry
up to 30x, then fail the trajectory with `empty_user_message`) is now
merged there (bxyu-nvidia/tau2-bench#20, NVIDIA-NeMo#17), superseding the personal
dev branches.

Verified end-to-end via NEL/EFB: 3x tau3-banking panda runs
(8.04/9.69/7.01% vs 8.66% reference, within run-to-run noise) and qwen
8.87% (matches AA 8.7). Empty user responses occurred and recovered
in-retry; zero hard-fails, zero ClientOSErrors.

---------

Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com>
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com>
Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants