Skip to content

feat(asb): add Agent Security Bench environment - #3567

Open
reinainblood wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
reinainblood:codex/asb-upstream-clean
Open

reinainblood wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
reinainblood:codex/asb-upstream-clean

Conversation

@reinainblood

@reinainblood reinainblood commented Sep 21, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds Agent Security Bench (ASB, ICLR 2025, arXiv:2410.02644) as a NeMo Gym environment: direct and observation prompt injection, memory poisoning, mixed attacks and plan-of-thought backdoors against tool-using agents, each run against its published defenses as well as undefended.

No issue is linked: this is benchmark onboarding rather than a change to existing behavior.

ASB does not distribute an evaluation dataset. Its runner constructs the benchmark at execution time by crossing agents, tasks, attacker tools, attack channels and defenses, so an integration has to recover the published expansion rather than point at a file. benchmarks/asb/prepare.py clones a pinned upstream revision, verifies every load-bearing input by SHA-256, and materializes the rows; benchmarks/asb/config.yaml exposes that as a benchmark with a prepare_script.

The denominator is 400 rows per condition — ten agents by one task by forty agent-matched attacker tools, with the PoT arm reaching the same figure through five agents by two tasks by forty. That contract is asserted in tests and recorded in a 27-condition, 10,800-row manifest. It was recovered from the granularity of the published percentages rather than assumed: defense-table cells quantize to 1/400, main-table DPI/OPI/MP cells to 1/2000, and Mixed only resolves on 1200, which is three attack types rather than five. main_attacker.py defaults task_num to 1 and the DPI/OPI/MP/mixed driver never overrides it.

Components:

  • resources_servers/asb — attack success, legitimate-task utility, judge-backed refusal, workflow-failure and memory-retrieval measurements, plus a /retrieve_memory route for the memory-poisoning conditions;
  • responses_api_agents/asb_agent — plan-then-execute loop matching ASB's, with attacker-tool injection, tool-call normalization and memory retrieval;
  • benchmarks/asb/reporting/report.py — the published table shape, averaging per-attack-type rates rather than pooling (Mixed's denominator is three attack types, so pooling would silently reweight it);
  • benchmarks/asb/reporting/graded.py — optional diagnostics, described below;
  • resources_servers/asb/task_data.py — row schema; required-ness tracks what the server and agent read by subscript, since verifier_metadata is untyped on the wire and a missing key raises mid-rollout rather than 422ing at validation.

No upstream source and no generated dataset is vendored. The prompt strings in benchmarks/asb/upstream_spec.py are transcribed verbatim under ASB's MIT license and recorded in ATTRIBUTIONS.md.

Getting the rows does not require generating them

Two supported paths, same 10,800 rows:

  • prepare materialize clones the pinned upstream revision, verifies each load-bearing input by SHA-256, and runs the expansion locally. Depends on nothing but upstream.
  • prepare pull fetches a frozen public copy of that expansion (10,800 rows at 1f561dccf92d, ASB's own MIT license). No credentials, nothing to configure.

prepare verify re-derives the rows from upstream and diffs the content hash, so the frozen copy is a convenience that can be checked rather than a source that has to be trusted.

pull defaults to that public copy, so reproducing a published number costs one command. push deliberately has no default and refuses to guess, because publishing into a namespace the caller did not name is a different kind of mistake from reading a public one. The copy is third-party rather than NeMo-owned and the docs say so; ASB_HF_REPO or --repo-id points either command elsewhere. If maintainers would prefer a NeMo-owned mirror, the rows are reproducible with a single command and I am glad to hand them over.

Disclosed deviations

Five compatibility deviations are listed in benchmarks/asb/METRICS.md and stay visible in generated reports:

  1. raise the generation cap from 256 to 4,096 tokens, because upstream's cap predates reasoning models and a reasoning trace exhausts it before any plan is emitted;
  2. rank the pinned poisoned-memory corpus lexically instead of with text-embedding-ada-002;
  3. attempt upstream's strict JSON parse first, then record any fenced or embedded-array salvage path;
  4. merge ASB's two consecutive system messages for templates that reject that layout, applied to every model identically;
  5. normalize parameterless tool declarations into the Responses API schema envelope.

Deviation 3 is the one that moves results, so METRICS.md quantifies it across four models rather than asserting it is small: salvage rates run from 0.0% to 40.2%, and a strict-parse sensitivity table shows what the worst-affected model's published cells would read without it. Deviations 1 and 4 are applied uniformly so cross-model comparisons stay internally valid.

The refusal judge's provider content-filters a small, non-random subset of aggressive observation-injection transcripts. Those rows are masked and reported as a measurement limitation rather than scored as model behavior; METRICS.md shows the loss is concentrated in OPI rather than spread.

Graded reporting

reporting.report renders ASB's published shape and its reward stays binary, because that is the benchmark's own definition: attack success is substring containment of the attack goal, utility is conjunctive over expected achievements.

reporting.graded is optional and separate. It reads per-step tool calls the rollouts already carry and reports when in a trajectory the attacker tool was first invoked, what fraction of a task's normal tools were invoked, and refusal as complied / refused / judge-reply-unparseable rather than two-valued. No re-run, no model calls.

Tool-invocation coverage is printed beside conjunctive utility specifically because the two do not share a measurement basis — utility is substring containment on response text, coverage reads invoked_tools — so a model can rank higher on one than the other in either direction. Where it does, that model's utility is partly measuring text emission rather than tool use. Nothing this module emits is an ASB metric or comparable to a published number, and the output says so.

Validation

  • Final branch synchronization (2026-10-01): merged upstream main at 31179922; pre-commit run --all-files and git diff --check passed on the resulting tree. GitHub DCO passed on the pushed head. Full NVIDIA CI is awaiting external-contributor vetting.
pre-commit run --all-files                                  passed, no hook modifications
pytest tests/unit_tests/                                    no new failures vs pristine main
pytest tests/unit_tests/test_task_data.py                   317 passed
pytest resources_servers/asb/tests responses_api_agents/asb_agent/tests   90 passed
gym env test +entrypoint=resources_servers/asb +should_validate_data=true          78 passed, data validated
gym env test +entrypoint=responses_api_agents/asb_agent +should_validate_data=true 12 passed
gym list benchmarks asb
gym eval prepare --benchmark asb                            10,800 rows across 27 conditions
gym dataset collate --resources-server asb --mode example_validation    5 rows, 5 unique ids
gym dataset collate --resources-server asb --mode train_preparation     10,800 rows, 10,800 unique ids

Schema validation was also run over the full matrix rather than the five committed example rows: all 10,800 validate against task_data.py with no failures.

Reproduction:

# regenerate from upstream ...
python -m benchmarks.asb.prepare materialize
# ... or pull the frozen public copy of the same rows, no credentials needed
python -m benchmarks.asb.prepare pull

python -m benchmarks.asb.prepare verify        # re-derive from upstream, diff the hash
gym eval prepare --benchmark asb

prepare verify is the check that establishes a pinned copy is upstream's, and is the one to run before trusting any number here.

Rollouts

resources_servers/asb/data/example_rollouts.jsonl carries five real rollouts, one per representative condition — clean control, direct prompt injection, observation prompt injection, memory poisoning, plan-of-thought backdoor — collected with openai/gpt-4o-mini as the policy through OpenRouter and the upstream-matched refusal judge. All five completed with zero masked rows and zero workflow failures.

The salvage-rate and judge-loss tables in METRICS.md are measured over a four-model, 43,177-rollout run of the full matrix.

Compatibility and benchmark impact

Adds new asb resources-server, asb_agent agent and asb benchmark entries. No existing environment, config or public API changes.

Running the full matrix requires materializing or restoring the pinned rows and configuring a policy model plus the gpt-4o-mini refusal judge. The server config resolves standalone without either, so gym eval prepare, gym dataset collate and gym list benchmarks work on a fresh checkout with no credentials.

Scores are comparable across runs of this adapter when the pinned manifest, disclosed deviations, judge route and model configuration are held fixed. They should not be read as numerically identical to the paper wherever a disclosed deviation changes the evaluated protocol.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

@copy-pr-bot

copy-pr-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 21, 2026
Agent Security Bench (ICLR 2025, arXiv:2410.02644) evaluates prompt injection,
observation injection, memory poisoning, mixed attacks and plan-of-thought
backdoors against tool-using agents, each run against its published defenses as
well as undefended.

ASB distributes no evaluation dataset. Its runner builds the benchmark at
execution time by crossing agents, tasks, attacker tools, attack channels and
defenses, so this adapter has to recover the published expansion rather than
point at a file. prepare.py clones a pinned upstream revision, verifies every
load-bearing input by SHA-256, and materializes the rows; config.yaml exposes
that as a benchmark with a prepare_script.

Every published condition is 400 rows: ten agents by one task by forty
agent-matched attacker tools, with PoT reaching the same figure through five
agents by two tasks by forty. That is recovered from the granularity of the
published percentages, not assumed -- defense-table cells quantize to 1/400,
main-table DPI/OPI/MP cells to 1/2000, and Mixed only resolves on 1200, which is
three attack types rather than five. 27 conditions, 10,800 rows.

Five compatibility deviations are disclosed in METRICS.md and stay visible in
generated reports: the raised output cap (upstream's 256 predates reasoning
models and is exhausted before a plan is emitted), lexical rather than ada-002
ranking of the poisoned memory corpus, plan-parse salvage recorded per row after
upstream's strict json.loads, merged consecutive system messages for templates
that reject ASB's two, and Responses API tool-schema normalization. Deviation 3
is quantified across four models rather than asserted small: salvage rates run
0.0% to 40.2%, with a strict-parse sensitivity table for the worst-affected one.

No upstream source and no generated dataset is vendored. The prompt strings in
upstream_spec.py are transcribed verbatim under ASB's MIT license and recorded
in ATTRIBUTIONS.md.

reporting.report renders the published table shape, averaging per-attack-type
rates rather than pooling, because Mixed's denominator is three attack types and
pooling would silently reweight it. reporting.graded is optional and separate:
it derives concession timing, normal-tool invocation coverage and three-valued
refusal from per-step tool calls the rollouts already carry, and states on its
face that none of it is comparable to a published number.

task_data.py makes required every field the server or agent reads by subscript.
verifier_metadata is untyped on the wire, so a malformed row 422s nowhere and
instead raises KeyError mid-rollout; marking those optional would move a crash
out of dataset validation and into the middle of a 10,800-row run.

Validation:
  pre-commit run --all-files                             passed, no modifications
  pytest tests/unit_tests/                               no new failures vs
                                                         pristine upstream/main
                                                         (20 pre-existing macOS
                                                         infra failures, all
                                                         reproduced on main)
  pytest tests/unit_tests/test_task_data.py              317 passed
  pytest resources_servers/asb responses_api_agents/asb_agent tests   90 passed
  gym env test +entrypoint=resources_servers/asb
      +should_validate_data=true                         78 passed, data validated
  gym env test +entrypoint=responses_api_agents/asb_agent
      +should_validate_data=true                         12 passed
  gym list benchmarks asb
  gym eval prepare --benchmark asb                       10,800 rows, 27 conditions
  gym dataset collate --mode example_validation          5 rows, 5 unique ids
  gym dataset collate --mode train_preparation           10,800 rows, 10,800 unique
  task_data schema validated over all 10,800 rows        0 failures

Rollouts: resources_servers/asb/data/example_rollouts.jsonl carries five real
rollouts, one per representative condition (clean control, DPI, OPI, memory
poisoning, PoT backdoor), collected with openai/gpt-4o-mini as policy through
OpenRouter and the upstream-matched refusal judge; all five completed with zero
masked rows and zero workflow failures. The salvage and judge-loss tables in
METRICS.md are measured over a four-model, 43,177-rollout run of the full matrix.

Signed-off-by: Kirsten Ruge <144049452+reinainblood@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@reinainblood
reinainblood force-pushed the codex/asb-upstream-clean branch from 6dceb9d to 0df01f9 Compare September 22, 2026 00:32
@reinainblood
reinainblood marked this pull request as ready for review September 22, 2026 02:06
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 22, 2026
Two review findings, both real.

The secrets detector fails on this PR. Reproduced with CI's own command --
detect-secrets-hook against .github/workflows/config/.secrets.baseline over the
merge-base diff -- which exits 1 on 18 Hex High Entropy String hits: the pinned
upstream commit sha, the SHA-256 of each upstream input file, the dataset content
hash, and the pinned Hub revision added below. Every one is reviewed and listed;
none is a credential. Baseline regenerated with the command the workflow prints,
which adds only these and removes nothing.

`prepare pull` advertised a frozen dataset and did not deliver one. It passed
revision=None, so it fetched whatever `main` pointed at, and checked nothing on
arrival. That repo's main has already moved once, for a documentation edit, and a
caller either side of it had no way to tell. It now pulls a pinned commit and
verifies the rows against the manifest that shipped with them, so a moved branch,
a partial download or a substituted copy raises instead of being scored. The
README no longer claims more than the code does.

Not changed: the benchmark manifest. `benchmarks/asb/manifest.yaml` is absent and
cannot currently be added honestly. The schema requires standard_prompt_config
for kind: benchmark, validation resolves a benchmark dataset's prompt through
`dataset.prompt_config or standard_prompt_config` and rejects a null result, and
ASB's rows carry a fully built responses_create_params.input. Supplying any value
makes preparation fail with "Some rows have responses_create_params.input but
prompt_config is also specified. These are mutually exclusive." Every benchmark
in the tree that ships pre-built prompts is in the same position, and none of
them carries a manifest. Raising separately rather than declaring a prompt
template this benchmark does not use.

Validation: 93 tests in resources_servers/asb and responses_api_agents/asb_agent,
including three new ones covering the pin and both verification failures;
tests/unit_tests/test_task_data.py 323 passed; pre-commit clean on all files;
`gym env test +entrypoint=resources_servers/asb +should_validate_data=true`
validates the data; the CI secrets command exits 0. A credential-free pull of the
pinned revision was run end to end and its content hash checked.

Signed-off-by: Kirsten Ruge <144049452+reinainblood@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@reinainblood
reinainblood requested a review from a team as a code owner September 22, 2026 07:08
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 23, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 23, 2026
@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Sep 23, 2026
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the benchmark-onboarding Self-contained benchmark onboarding label Oct 21, 2026
reinainblood and others added 2 commits October 27, 2026 17:58
Upstream rollout dispatch now resolves benchmarks through environment_servers
(NVIDIA-NeMo#3482); without a declaration gym eval run --benchmark asb cannot route rows
to the ASB agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
…clean

Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>

# Conflicts:
#	.github/workflows/config/.secrets.baseline
#	ATTRIBUTIONS.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers benchmark-onboarding Self-contained benchmark onboarding community-request Issue reported or requested by someone from the community complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work sla:triage-overdue Review assignment is over the one-business-day SLA waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants