Repository navigation
feat(asb): add Agent Security Bench environment - #3567
Open
reinainblood wants to merge 5 commits into
Open
reinainblood wants to merge 5 commits into
reinainblood wants to merge 5 commits into
Conversation
Agent Security Bench (ICLR 2025, arXiv:2410.02644) evaluates prompt injection,
observation injection, memory poisoning, mixed attacks and plan-of-thought
backdoors against tool-using agents, each run against its published defenses as
well as undefended.
ASB distributes no evaluation dataset. Its runner builds the benchmark at
execution time by crossing agents, tasks, attacker tools, attack channels and
defenses, so this adapter has to recover the published expansion rather than
point at a file. prepare.py clones a pinned upstream revision, verifies every
load-bearing input by SHA-256, and materializes the rows; config.yaml exposes
that as a benchmark with a prepare_script.
Every published condition is 400 rows: ten agents by one task by forty
agent-matched attacker tools, with PoT reaching the same figure through five
agents by two tasks by forty. That is recovered from the granularity of the
published percentages, not assumed -- defense-table cells quantize to 1/400,
main-table DPI/OPI/MP cells to 1/2000, and Mixed only resolves on 1200, which is
three attack types rather than five. 27 conditions, 10,800 rows.
Five compatibility deviations are disclosed in METRICS.md and stay visible in
generated reports: the raised output cap (upstream's 256 predates reasoning
models and is exhausted before a plan is emitted), lexical rather than ada-002
ranking of the poisoned memory corpus, plan-parse salvage recorded per row after
upstream's strict json.loads, merged consecutive system messages for templates
that reject ASB's two, and Responses API tool-schema normalization. Deviation 3
is quantified across four models rather than asserted small: salvage rates run
0.0% to 40.2%, with a strict-parse sensitivity table for the worst-affected one.
No upstream source and no generated dataset is vendored. The prompt strings in
upstream_spec.py are transcribed verbatim under ASB's MIT license and recorded
in ATTRIBUTIONS.md.
reporting.report renders the published table shape, averaging per-attack-type
rates rather than pooling, because Mixed's denominator is three attack types and
pooling would silently reweight it. reporting.graded is optional and separate:
it derives concession timing, normal-tool invocation coverage and three-valued
refusal from per-step tool calls the rollouts already carry, and states on its
face that none of it is comparable to a published number.
task_data.py makes required every field the server or agent reads by subscript.
verifier_metadata is untyped on the wire, so a malformed row 422s nowhere and
instead raises KeyError mid-rollout; marking those optional would move a crash
out of dataset validation and into the middle of a 10,800-row run.
Validation:
pre-commit run --all-files passed, no modifications
pytest tests/unit_tests/ no new failures vs
pristine upstream/main
(20 pre-existing macOS
infra failures, all
reproduced on main)
pytest tests/unit_tests/test_task_data.py 317 passed
pytest resources_servers/asb responses_api_agents/asb_agent tests 90 passed
gym env test +entrypoint=resources_servers/asb
+should_validate_data=true 78 passed, data validated
gym env test +entrypoint=responses_api_agents/asb_agent
+should_validate_data=true 12 passed
gym list benchmarks asb
gym eval prepare --benchmark asb 10,800 rows, 27 conditions
gym dataset collate --mode example_validation 5 rows, 5 unique ids
gym dataset collate --mode train_preparation 10,800 rows, 10,800 unique
task_data schema validated over all 10,800 rows 0 failures
Rollouts: resources_servers/asb/data/example_rollouts.jsonl carries five real
rollouts, one per representative condition (clean control, DPI, OPI, memory
poisoning, PoT backdoor), collected with openai/gpt-4o-mini as policy through
OpenRouter and the upstream-matched refusal judge; all five completed with zero
masked rows and zero workflow failures. The salvage and judge-loss tables in
METRICS.md are measured over a four-model, 43,177-rollout run of the full matrix.
Signed-off-by: Kirsten Ruge <144049452+reinainblood@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reinainblood
force-pushed
the
codex/asb-upstream-clean
branch
from
September 22, 2026 00:32
6dceb9d to
0df01f9
Compare
reinainblood
marked this pull request as ready for review
September 22, 2026 02:06
Two review findings, both real. The secrets detector fails on this PR. Reproduced with CI's own command -- detect-secrets-hook against .github/workflows/config/.secrets.baseline over the merge-base diff -- which exits 1 on 18 Hex High Entropy String hits: the pinned upstream commit sha, the SHA-256 of each upstream input file, the dataset content hash, and the pinned Hub revision added below. Every one is reviewed and listed; none is a credential. Baseline regenerated with the command the workflow prints, which adds only these and removes nothing. `prepare pull` advertised a frozen dataset and did not deliver one. It passed revision=None, so it fetched whatever `main` pointed at, and checked nothing on arrival. That repo's main has already moved once, for a documentation edit, and a caller either side of it had no way to tell. It now pulls a pinned commit and verifies the rows against the manifest that shipped with them, so a moved branch, a partial download or a substituted copy raises instead of being scored. The README no longer claims more than the code does. Not changed: the benchmark manifest. `benchmarks/asb/manifest.yaml` is absent and cannot currently be added honestly. The schema requires standard_prompt_config for kind: benchmark, validation resolves a benchmark dataset's prompt through `dataset.prompt_config or standard_prompt_config` and rejects a null result, and ASB's rows carry a fully built responses_create_params.input. Supplying any value makes preparation fail with "Some rows have responses_create_params.input but prompt_config is also specified. These are mutually exclusive." Every benchmark in the tree that ships pre-built prompts is in the same position, and none of them carries a manifest. Raising separately rather than declaring a prompt template this benchmark does not use. Validation: 93 tests in resources_servers/asb and responses_api_agents/asb_agent, including three new ones covering the pin and both verification failures; tests/unit_tests/test_task_data.py 323 passed; pre-commit clean on all files; `gym env test +entrypoint=resources_servers/asb +should_validate_data=true` validates the data; the CI secrets command exits 0. A credential-free pull of the pinned revision was run end to end and its content hash checked. Signed-off-by: Kirsten Ruge <144049452+reinainblood@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
This was referenced Oct 22, 2026
Upstream rollout dispatch now resolves benchmarks through environment_servers (NVIDIA-NeMo#3482); without a declaration gym eval run --benchmark asb cannot route rows to the ASB agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
…clean Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com> # Conflicts: # .github/workflows/config/.secrets.baseline # ATTRIBUTIONS.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Agent Security Bench (ASB, ICLR 2025, arXiv:2410.02644) as a NeMo Gym environment: direct and observation prompt injection, memory poisoning, mixed attacks and plan-of-thought backdoors against tool-using agents, each run against its published defenses as well as undefended.
No issue is linked: this is benchmark onboarding rather than a change to existing behavior.
ASB does not distribute an evaluation dataset. Its runner constructs the benchmark at execution time by crossing agents, tasks, attacker tools, attack channels and defenses, so an integration has to recover the published expansion rather than point at a file.
benchmarks/asb/prepare.pyclones a pinned upstream revision, verifies every load-bearing input by SHA-256, and materializes the rows;benchmarks/asb/config.yamlexposes that as a benchmark with aprepare_script.The denominator is 400 rows per condition — ten agents by one task by forty agent-matched attacker tools, with the PoT arm reaching the same figure through five agents by two tasks by forty. That contract is asserted in tests and recorded in a 27-condition, 10,800-row manifest. It was recovered from the granularity of the published percentages rather than assumed: defense-table cells quantize to 1/400, main-table DPI/OPI/MP cells to 1/2000, and Mixed only resolves on 1200, which is three attack types rather than five.
main_attacker.pydefaultstask_numto 1 and the DPI/OPI/MP/mixed driver never overrides it.Components:
resources_servers/asb— attack success, legitimate-task utility, judge-backed refusal, workflow-failure and memory-retrieval measurements, plus a/retrieve_memoryroute for the memory-poisoning conditions;responses_api_agents/asb_agent— plan-then-execute loop matching ASB's, with attacker-tool injection, tool-call normalization and memory retrieval;benchmarks/asb/reporting/report.py— the published table shape, averaging per-attack-type rates rather than pooling (Mixed's denominator is three attack types, so pooling would silently reweight it);benchmarks/asb/reporting/graded.py— optional diagnostics, described below;resources_servers/asb/task_data.py— row schema; required-ness tracks what the server and agent read by subscript, sinceverifier_metadatais untyped on the wire and a missing key raises mid-rollout rather than 422ing at validation.No upstream source and no generated dataset is vendored. The prompt strings in
benchmarks/asb/upstream_spec.pyare transcribed verbatim under ASB's MIT license and recorded inATTRIBUTIONS.md.Getting the rows does not require generating them
Two supported paths, same 10,800 rows:
prepare materializeclones the pinned upstream revision, verifies each load-bearing input by SHA-256, and runs the expansion locally. Depends on nothing but upstream.prepare pullfetches a frozen public copy of that expansion (10,800 rows at1f561dccf92d, ASB's own MIT license). No credentials, nothing to configure.prepare verifyre-derives the rows from upstream and diffs the content hash, so the frozen copy is a convenience that can be checked rather than a source that has to be trusted.pulldefaults to that public copy, so reproducing a published number costs one command.pushdeliberately has no default and refuses to guess, because publishing into a namespace the caller did not name is a different kind of mistake from reading a public one. The copy is third-party rather than NeMo-owned and the docs say so;ASB_HF_REPOor--repo-idpoints either command elsewhere. If maintainers would prefer a NeMo-owned mirror, the rows are reproducible with a single command and I am glad to hand them over.Disclosed deviations
Five compatibility deviations are listed in
benchmarks/asb/METRICS.mdand stay visible in generated reports:text-embedding-ada-002;Deviation 3 is the one that moves results, so
METRICS.mdquantifies it across four models rather than asserting it is small: salvage rates run from 0.0% to 40.2%, and a strict-parse sensitivity table shows what the worst-affected model's published cells would read without it. Deviations 1 and 4 are applied uniformly so cross-model comparisons stay internally valid.The refusal judge's provider content-filters a small, non-random subset of aggressive observation-injection transcripts. Those rows are masked and reported as a measurement limitation rather than scored as model behavior;
METRICS.mdshows the loss is concentrated in OPI rather than spread.Graded reporting
reporting.reportrenders ASB's published shape and its reward stays binary, because that is the benchmark's own definition: attack success is substring containment of the attack goal, utility is conjunctive over expected achievements.reporting.gradedis optional and separate. It reads per-step tool calls the rollouts already carry and reports when in a trajectory the attacker tool was first invoked, what fraction of a task's normal tools were invoked, and refusal as complied / refused / judge-reply-unparseable rather than two-valued. No re-run, no model calls.Tool-invocation coverage is printed beside conjunctive utility specifically because the two do not share a measurement basis — utility is substring containment on response text, coverage reads
invoked_tools— so a model can rank higher on one than the other in either direction. Where it does, that model's utility is partly measuring text emission rather than tool use. Nothing this module emits is an ASB metric or comparable to a published number, and the output says so.Validation
mainat31179922;pre-commit run --all-filesandgit diff --checkpassed on the resulting tree. GitHub DCO passed on the pushed head. Full NVIDIA CI is awaiting external-contributor vetting.Schema validation was also run over the full matrix rather than the five committed example rows: all 10,800 validate against
task_data.pywith no failures.Reproduction:
prepare verifyis the check that establishes a pinned copy is upstream's, and is the one to run before trusting any number here.Rollouts
resources_servers/asb/data/example_rollouts.jsonlcarries five real rollouts, one per representative condition — clean control, direct prompt injection, observation prompt injection, memory poisoning, plan-of-thought backdoor — collected withopenai/gpt-4o-minias the policy through OpenRouter and the upstream-matched refusal judge. All five completed with zero masked rows and zero workflow failures.The salvage-rate and judge-loss tables in
METRICS.mdare measured over a four-model, 43,177-rollout run of the full matrix.Compatibility and benchmark impact
Adds new
asbresources-server,asb_agentagent andasbbenchmark entries. No existing environment, config or public API changes.Running the full matrix requires materializing or restoring the pinned rows and configuring a policy model plus the
gpt-4o-minirefusal judge. The server config resolves standalone without either, sogym eval prepare,gym dataset collateandgym list benchmarkswork on a fresh checkout with no credentials.Scores are comparable across runs of this adapter when the pinned manifest, disclosed deviations, judge route and model configuration are held fixed. They should not be read as numerically identical to the paper wherever a disclosed deviation changes the evaluated protocol.
Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).