Repository navigation
fix: tau2 knowledge deps + repoint tau2-bench pins to stable branches - #1954
Merged
Merged
Conversation
Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com>
cmunley1
previously approved these changes
Jul 8, 2026
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
… info counters Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
…eeps secrets-detector quiet) Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
wprazuch
approved these changes
Jul 9, 2026
wprazuch
left a comment
Contributor
There was a problem hiding this comment.
Verified end-to-end: 4x tau3-banking full runs via NEL/EFB against these pins (3x panda within noise of reference, qwen matching AA), tau2 tests green against bxyu/nemo_gym_stable.
laszkiewiczp
added a commit
that referenced
this pull request
Jul 10, 2026
The tau2-bench pin tracks a branch that moved schema (nl_assertions), so the recorded request/response fixture on this base no longer matches what the server produces; take the updated fixture from main's fix commit f245781 without its runtime changes, which do not apply to this base's tau2 layout. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com>
hkumar92
pushed a commit
to hkumar92/Gym
that referenced
this pull request
Jul 14, 2026
The tau2-bench pin tracks a branch that moved schema (nl_assertions), so the recorded request/response fixture on this base no longer matches what the server produces; take the updated fixture from main's fix commit f245781 without its runtime changes, which do not apply to this base's tau2 layout. Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Signed-off-by: Haran Kumar Shiv Kumar <hshivkumar@nvidia.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Jul 15, 2026
…NVIDIA-NeMo#1954) Two fixes needed to run tau3 through NEL: - Pin the tau2 `[knowledge]` extra deps (rank-bm25, openai) explicitly in requirements.txt — the extra is not applied to a `tau2[knowledge] @ git+URL` direct reference, so bm25/bm25_grep environments miss rank-bm25 on install. - Repoint the tau2-bench code pin (requirements.txt, source.py default, README) to `bxyu/nemo_gym_stable` and the data ref mention to `bxyu/nemo_gym_data` — the empty-user-message hard-fail handling (retry up to 30x, then fail the trajectory with `empty_user_message`) is now merged there (bxyu-nvidia/tau2-bench#20, NVIDIA-NeMo#17), superseding the personal dev branches. Verified end-to-end via NEL/EFB: 3x tau3-banking panda runs (8.04/9.69/7.01% vs 8.66% reference, within run-to-run noise) and qwen 8.87% (matches AA 8.7). Empty user responses occurred and recovered in-retry; zero hard-fails, zero ClientOSErrors. --------- Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com> Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes needed to run tau3 through NEL:
[knowledge]extra deps (rank-bm25, openai) explicitly in requirements.txt — the extra is not applied to atau2[knowledge] @ git+URLdirect reference, so bm25/bm25_grep environments miss rank-bm25 on install.bxyu/nemo_gym_stableand the data ref mention tobxyu/nemo_gym_data— the empty-user-message hard-fail handling (retry up to 30x, then fail the trajectory withempty_user_message) is now merged there (Reapply user-empty-message retries + hard-fail trajectories (supersedes #16) bxyu-nvidia/tau2-bench#20, AzureOpenAIModel implementation #17), superseding the personal dev branches.Verified end-to-end via NEL/EFB: 3x tau3-banking panda runs (8.04/9.69/7.01% vs 8.66% reference, within run-to-run noise) and qwen 8.87% (matches AA 8.7). Empty user responses occurred and recovered in-retry; zero hard-fails, zero ClientOSErrors.