Skip to content

Refuse calls on Connections with an unenforced caller_assertion - #556

Merged
Antnation merged 6 commits into
mainfrom
fix/463-caller-assertion-fail-closed
Oct 15, 2026
Merged

Antnation merged 6 commits into
mainfrom
fix/463-caller-assertion-fail-closed

Conversation

@Antnation

@Antnation Antnation commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Part of #463

Problem

Since #480 the Connection schemas and admin API accept and persist a caller_assertion block, but nothing outside its own module reads it. A Connection that carries one is served exactly like one that does not: the upstream gets no assertion header, and calls that arrive without a principal are not refused. For an operator who configured "every call names its caller" that is a silent fail-open.

What this changes

One refusal, before anything else happens. All five target constructors in ConnectionHttpRuntime (target, mcp_target, openapi_discovery_target, test_target, mcp_test_target) now read their record through a single managed_record helper that returns ConnectionHttpError::CallerAssertionNotEnforced when caller_assertion is set. A ConnectionHttpTarget is the only way any lane reaches an egress client, a TLS identity or a credential, so this covers:

  • tool calls (manual and OpenAPI-generated HTTP tools, composite steps)
  • proxied requests on Connection-bound routes
  • managed MCP tool calls, catalog discovery and protocol probes
  • OpenAPI preview, register and refresh
  • enum and label source fetches
  • stored HTTP and MCP Connection tests (including for a disabled Connection)
  • tool binding validation (validate_binding) and proxy route construction, which reject a binding to such a Connection just as they reject a binding to a disabled one

I put the check in target construction rather than in resolve_credential / resolve_credential_before. Every lane runs the egress preflight (DNS) and prepare_transport (TLS secret reads) before it resolves the credential. A check inside credential resolution would therefore come after egress-adjacent work, and it would not cover lanes that skip credentials (use_connection_authentication: false). The refusal at target construction comes before DNS, before any secret read, before OAuth minting and before any upstream byte.

One fixed reason: caller_assertion_not_enforced. Where it shows up:

Surface Where it shows up
Tool calls tool.invoke_failure with reason: work_error, failure_reason: caller_assertion_not_enforced (#444 path); tool.upstream_request failure carries the same reason
Composite tools failure_reason: composite_failed in the audit event; the step's caller_assertion_not_enforced is in the error details returned to the caller only (unchanged composite behaviour)
Connection tests new ConnectionTestReason::CallerAssertionNotEnforced on the authenticated stage, status request_failed (HTTP and MCP probe identical); the admin UI accepts the new reason
OpenAPI / MCP catalog new OpenApiCatalogError / McpCatalogRefreshError variants, HTTP 409 like a disabled Connection
Enum sources new EnumSourceFailureReason::CallerAssertionNotEnforced in connection.enum_source_refresh_failed (it used to fall through to egress_denied)
Proxy 502 bad_gateway; the reason is only error_category on the existing fail-closed warning log line, not a structured audit event, as for every other Connection failure on a proxy route

No key material is read, so none can leak. The executor test writes canary values behind both the API-key alias and the assertion signing-key alias, checks that neither shows up in the caller error, any audit event or the proxy log, and then repeats the call with both secret files removed. A provider read before the refusal would then fail as credential_unavailable and raise connection.secret_resolution_failed. The reason stays caller_assertion_not_enforced and no such event appears.

Keeping the refusal in place.

  • Admin edits. The admin UI does not model caller_assertion, and the safe read hides its signing key alias, so the UI cannot send the block back. The replace handler read the missing key as removal, so a routine UI save (display name, enabled flag, secret rotation) stripped the block and put the Connection back into service unattested. A PUT that omits caller_assertion now keeps the current block, as omitted hidden secret bindings already are. An explicit "caller_assertion": null removes it, and the replace request schema accepts that null.
  • Live OpenAPI catalogs. The tool registry runs binding validation over the merged definitions of every lane on every change. A live catalog on a Connection that gains the block would have made every later registry change fail (TOOLS_FILE reloads, other Connections' refreshes). A catalog registered on a Connection that already had the block (register allowed it before this PR) would have stopped startup after an upgrade. Such a catalog is now withdrawn from the registry and kept in storage, exactly like a disabled Connection's: at boot, in the authority reconcile and on Connection change.

Validation (validate_caller_assertion).

  • bind_approval: true is rejected with caller_assertion.bind_approval / unsupported_binding. No approval subsystem exists, so the setting could only ever be unsatisfiable or ignored. The published request schemas pin it to const: false.
  • caller_assertion.header_name must differ, case-insensitively, from every additional_headers[].header_name (duplicate_header_name). The existing primary-credential check is unchanged.
  • Both rules apply to new and replacement writes only. validated_persisted_v0 keeps documents written before these rules readable, the same way it handles the legacy OPTIONS test profile. Without that, one such row would make the store list fail and could stop the gateway from starting after an upgrade. Those documents are still refused at runtime, like every Connection with an assertion.

RS256 round trip. caller_assertion_tests.rs generates an RSA-2048 key at test runtime, signs with it, and verifies with jsonwebtoken using only the JWK that rs256_public_jwk publishes. It also checks that another key's JWK refuses the token and that the RSA key will not load as ES256. No PEM is committed. Generating the key needs aws-lc-rs as a dev-dependency. It is already in the graph as jsonwebtoken's backend at the same 1.18.1, so this adds a dev-only dependency edge on gateway, not a new crate, and no package's features change. rcgen is built on ring here and cannot generate RSA keys, and jsonwebtoken has no key generation.

Docs. The operator guide's "Signed caller assertions" section says the block is accepted but not yet enforced, that Connections carrying it are refused until request-path injection ships, where the reason appears for each lane, the binding and catalog consequences, and how to remove the block. Both schemas' descriptions lead with that status, and the CHANGELOG Security entry carries an upgrade note.

Upgrade note. A TOOLS_FILE tool or proxy route bound to a Connection that already carries the block (allowed since #480) now fails startup or the reload that contains it, not just that binding, the same as a binding to a disabled Connection.

Rebase

Rebased onto main at 5d99076 (#549: Rust 1.99.0 / Node 26.10.0, jsonwebtoken 11.1.0, jsonschema 0.56, rustls 0.23.46). The CHANGELOG conflict keeps both Security entries. transport-ownership.json was checked against the merged lockfile: the guard's dependency-review stage (graph and build_inputs equality with cargo metadata --locked --all-features) passes. Compared with main, the only graph change is the dev-only aws-lc-rs@1.18.1 edge on gateway, and scopes and build inputs are identical. The syntax stage needs the gateway build and runs in CI.

Not in this PR (remaining #463 acceptance)

Not here: minting and injecting the assertion on the request path, refusing principal-less calls once injection exists, the JWKS endpoint and its rotation procedure, bind_approval once an approval subsystem exists, and admin UI editing of the block. When injection lands, it has to lift the refusal added here deliberately. It also needs to decide whether an assertion should require HTTPS: requires_tls does not consider caller_assertion today. That is harmless while nothing is sent, but it matters once a bearer assertion is. Also left for follow-up: requires_secrets_write_to_replace does not treat a change to caller_assertion.signing_key_id as credential-authority work.

Testing

  • Rust (CI; the gateway crate cannot build on my machine because build.rs pins Node exactly):
    • connections::http: every HTTP and MCP target constructor refuses, ConnectionTestService reports the reason with no client prepared, and removing the block restores the target
    • tools::executor_tests: refused tool call with audit failure_reason, no canary in errors or audit, no upstream request, and a repeat with the secret files removed proving the provider is never read; composite leaf refused before member I/O with the leaf reason in details
    • proxy::forward: refused proxied request, fixed 502 outcome with no attempts, error_category in the captured log, no canaries
    • connections::openapi: preview, register and refresh refused before any fetch; a live catalog is withdrawn when its Connection gains the block, and the registry still validates with the gateway's binding validator
    • connections::mcp: refresh refused before any session; session-call reason tied to the typed error
    • app_tests: PUT keeps an omitted block and removes it on null; catalog refusals are 409 with the reason
    • tools::enum_source: refusal after policy and before egress
    • connections::test and tools::mcp_upstream: classification parity between HTTP and MCP
    • connections::model: both validation rules, schema const, persisted readability
    • connections::caller_assertion: the RS256 round trip
  • cargo fmt --all --check with the CI toolchain
  • python scripts/transport_guard.py check: dependency review passes against the merged lockfile (syntax stage in CI)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T01:48:50.205184Z 411dec7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 411dec743e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.managed()
.get(connection_id)
.ok_or(ConnectionHttpError::ConnectionNotFound)?;
refuse_unenforced_caller_assertion(record)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Filter refused OpenAPI catalogs before registry validation

When upgrading a database that already contains both a caller_assertion Connection and a nonempty registered OpenAPI catalog—both accepted by the previous release—the boot loader still classifies that catalog as active in OpenApiConnectionCatalogService::load_with_enum_sources. Its definitions then enter the registry validator, which calls validate_binding; this new refusal makes the merge return an error and aborts gateway startup. Treat these catalogs like disabled-Connection catalogs by excluding or withdrawing them before registry validation, while retaining the durable catalog for later reactivation.

Useful? React with 👍 / 👎.

| ConnectionHttpError::ConnectionNotFound
| ConnectionHttpError::WrongConnectionKind
| ConnectionHttpError::UnsupportedAuthentication
| ConnectionHttpError::CallerAssertionNotEnforced

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Audit caller-assertion proxy refusals

When a connection-bound proxy request hits this new branch, the refusal reason is emitted only through tracing::warn; the UpstreamOutcome attached to the response has no reason field, so the structured http.request_observed audit records only a generic 502 with zero attempts and cannot identify the caller-assertion security decision. Emit a dedicated audit event or propagate the bounded reason into the structured request audit.

AGENTS.md reference: AGENTS.md:L55-L55

Useful? React with 👍 / 👎.

The Connection model and admin API have accepted and persisted a
caller_assertion block since it was added, but nothing on the request
path mints or injects the assertion. A Connection carrying one was served
exactly like one without it, so the upstream received calls attested by
nobody, including calls that arrived with no principal at all. That is a
silent fail-open for an operator who configured every call to name its
caller.

Refuse such a Connection where every lane gets its target. All five
target constructors now read the record through one helper that rejects
a caller_assertion with CallerAssertionNotEnforced, and a
ConnectionHttpTarget is the only way tool calls, the proxy, managed MCP
sessions and discovery, OpenAPI discovery, enum and label sources and
Connection tests reach an egress client, TLS identity or credential. The
refusal therefore lands before DNS, secret reads, OAuth minting and any
upstream byte, and no lane can route around it. Placing it in credential
resolution instead would have been too late: every lane runs the egress
preflight and resolves TLS material before it resolves the credential.

The refusal keeps the fixed reason caller_assertion_not_enforced on every
surface: the tool work-failure audit (#444 failure_reason), the Connection
test stage reason (also accepted by the admin UI), the OpenAPI and MCP
catalog errors (409), enum source failures, and the proxy's error
category (502).

New and replacement writes also reject bind_approval true, since there is
no approval subsystem a mutation could be bound to, and an assertion
header_name equal to any additional header name in any case. Documents
persisted before those rules stay readable through the v0 persisted
reader, as legacy OPTIONS test profiles do, so an upgrade cannot turn the
store unreadable; they are refused at runtime like every Connection that
carries an assertion. The published schemas and the operator guide say
the block is accepted but not yet enforced.
Only ES256 had a sign/verify test. RS256 needs an RSA key, and no private
key belongs in this tree, but neither rcgen (built on ring here) nor
jsonwebtoken can generate one. aws-lc-rs can, and it is already in the
graph as jsonwebtoken's backend at the same 1.18.1, so it becomes a
dev-dependency edge rather than a new crate; rcgen only wraps the
generated PKCS#8 DER as PEM.

The test signs with the generated key and verifies with a standard
verifier holding nothing but the JWK this gateway publishes for it, which
also covers rs256_public_jwk. A second generated key's JWK must refuse the
token, and the RSA key must not load as an ES256 key.

The transport-ownership graph records the new dev-dependency edge; no
crate, feature or syntax scope changed.
The admin UI does not model caller_assertion, and the safe read replaces
its signing key alias with signing_key_configured, so the UI cannot send
the block back. The replace handler read the missing key as removal, so
any routine save from the UI (a display name, the enabled flag, a secret
rotation) stripped the block. That lifted the caller_assertion_not_enforced
refusal and put the Connection back into service with no assertion: the
same silent fail-open #463 is about.

A replacement that omits the key now keeps the current block, the way
omitted hidden secret bindings are already kept. Removing the block takes
an explicit "caller_assertion": null, and the replace request schema now
accepts that null. A Connection that never had the block is unaffected.
validate_binding now refuses a Connection with a caller_assertion, and the
tool registry runs that binding validation over the merged definitions of
every lane on every change. Without this change, a live managed OpenAPI
catalog on such a Connection would break more than its own tools. After an
admin added the block to a Connection with a registered catalog, every
later registry change (a TOOLS_FILE reload, another Connection's OpenAPI or
MCP refresh) would fail validation. A catalog registered on a Connection
that already had the block, which register allowed before this branch,
would stop the gateway from starting after an upgrade, because the boot
merge would refuse it.

Such a catalog is now withdrawn from the registry and kept in storage, as a
disabled Connection's catalog already is: at boot, in the authority
reconcile and when the Connection changes. Its tools would be refused on
every call anyway. Tests cover the withdrawal against the real binding
validator, and the refusal of preview, register and refresh with
caller_assertion_not_enforced before any spec fetch.
Every lane was already refused at target construction, but several
surfaces were only checked for "some error" or for a missing event, which
would pass for the wrong reason too:

- MCP catalog refresh mapped the target error by matching reason strings
  and fell back to discovery_not_configured on anything it did not know,
  so a renamed reason would have changed the result silently. It now
  matches the error variant. Tests cover refresh on an attested Connection
  and tie the session-call reason string to the typed error.
- OpenAPI and MCP catalog refusals are asserted as 409 with
  caller_assertion_not_enforced.
- The proxy test asserted only a 502 with no attempts, which any target
  error produces. It now captures the fail-closed log line, the only place
  the proxy records why, and checks it names the refusal and leaks no
  secret.
- The executor test checked for no secret-resolution event while both
  secrets still resolved, so the event was absent either way. A second
  call now runs after both secret files are removed. A provider read before
  the refusal would then fail as credential_unavailable and raise that
  event. The reason stays caller_assertion_not_enforced and the event never
  appears.
- A composite leaf on such a Connection is refused before member I/O. The
  invocation reports composite_failed, and the leaf reason reaches the
  caller's details.
The operator guide and CHANGELOG said every refusal carries the reason in
the tool audit failure_reason. Two lanes do not. A composite tool's audit
failure_reason is composite_failed, and the step's reason reaches only the
error details returned to the caller. A proxied request records the reason
only as error_category on the gateway's warning log line, as for every
other Connection failure on a proxy route. Both now say so, lane by lane.

They also now cover the upgrade impact: a TOOLS_FILE tool or proxy route
bound to such a Connection fails startup or the reload that contains it,
the same as a binding to a disabled Connection. They describe the OpenAPI
catalog withdrawal and the PUT rule that keeps an omitted block. The
schema descriptions lead with the fact that the block is not yet enforced
and describe signing as what will happen, not as what happens today.
@Antnation
Antnation force-pushed the fix/463-caller-assertion-fail-closed branch from 411dec7 to 13bf909 Compare October 15, 2026 02:41
@Antnation
Antnation merged commit 6049f44 into main Oct 15, 2026
35 checks passed
Antnation added a commit that referenced this pull request Oct 15, 2026
@Antnation
Antnation deleted the fix/463-caller-assertion-fail-closed branch October 15, 2026 03:15
Antnation added a commit that referenced this pull request Oct 20, 2026
Main's #566 rewrote the caller-assertion refusal tests to serve the API
key and signing key from a counting provider, which proves the #556
refusal never consults a secret provider. That proof is kept. The
signing key it serves is now a generated PKCS#8 key, and the fixture
saves the Connection through the same deferred-binding pass the admin
API runs, so the key is really loaded. The tests assert that the save
read each secret exactly once, and that the refused call and the
composite leaf read nothing more.

Both schemas take main's opening for the caller_assertion block and
describe the keys this branch now resolves and publishes.
Antnation added a commit that referenced this pull request Oct 20, 2026
Releases since #556 stored an enabled Connection's caller_assertion
without reading its signing key, so a stored document can name an alias
that is unconfigured, a placeholder, or not a PKCS#8 key. Checking those
keys at load made ConnectionControlPlane::from_config refuse to start and
made a cluster reconcile fail for the whole snapshot, which takes every
other Connection down with one document.

Keys are now checked apart from credentials when a stored document is
loaded. Startup and reconcile still refuse a credential this replica
cannot resolve, as before, but a caller-assertion key that cannot be used
leaves only that Connection's keys out of the JWKS. It is logged with the
Connection, field and code, and reported as
configuration.caller_assertion.key_problem on the admin detail. The #556
refusal is unchanged, so the Connection stays refused on every lane. A
save is still held to the write-time rule and refused with the existing
safe codes until the key is usable; disabling needs no key.

The JWKS build follows the same rule: a Connection publishes all of its
keys or none. One whose key cannot be read or loaded, or that publishes a
kid another Connection publishes for a different key, is left out, and
the rest of the set is still published rather than the whole set going
503. A set that left a Connection out is rebuilt after 30 seconds instead
of 300.

The build no longer holds the cache lock across secret resolution. The
cache sits behind a short std lock, a separate async lock serializes
builds, and a fetch that arrives during a rebuild for the same keys is
answered from the previous set instead of queueing on the provider. A
fetch with nothing it may serve waits at most 10 seconds and then gets
503 no-store. The snapshot is read again once the build lock is held, so
a waiter can no longer rebuild from an older snapshot and republish a
withdrawn key. Each alias is resolved once per build. After a failed
refresh the gateway does not keep serving the last good keys: what an
unpinned alias holds may have changed, and the gateway does not use such
an alias's material as a stale fallback.

The PEM check now counts every BEGIN marker in the material, because the
parser skips blocks whose label it does not know. The kid-uniqueness
comment now says the write check is per replica and the JWKS is the
backstop.

The admin-router caller-assertion tests from #566 now bind a real key
and create with secrets-write, since a caller assertion is credential
authority. They assert that a routine rename by an editor still passes
and that removing the block without secrets-write is refused.
Antnation added a commit that referenced this pull request Oct 21, 2026
Connections carrying a caller_assertion block have been refused on every
lane since #556 because nothing signed or injected the assertion. A manual
HTTP tool call is the lane that has both a validated principal and the
exact request in hand before it is sent, so it now attests its caller:

- ToolInvocationContext carries an AttestedCaller built from the
  authenticated auth::Principal where the call enters (MCP endpoint, admin
  playground), never from the audit actor. Service tokens and principals
  without an issuer yield none.
- ConnectionHttpRuntime::tool_call_target is the only constructor that
  serves an attested Connection, and only when the stored block passes the
  rules a new write is held to (bind_approval false, no header collision,
  HTTPS). Every other constructor keeps refusing with
  caller_assertion_not_enforced, so proxied requests, managed MCP,
  discovery, enum sources, stored tests, OpenAPI and composite tools are
  unchanged.
- A call with no principal is refused with caller_assertion_no_principal
  before the egress preflight, any secret read or any upstream byte; a
  Connection whose kid another enabled Connection publishes for a different
  key is refused with caller_assertion_key_not_published.
- Per call the signing key is resolved and loaded through the same path as
  the save and the JWKS, and one assertion is signed after every other
  header and the body are final: jti is the request id, iat/exp follow
  lifetime_seconds, and bind_request covers the method, the origin-form
  request target and the SHA-256 of the exact body bytes. It is written as
  the last header and refuses rather than replaces an existing value.
- Refusals use distinct fixed reasons through the #444 failure_reason
  path; key read failures also raise connection.secret_resolution_failed.
  tool.upstream_request records the assertion's kid and jti, never the
  token or key material.
- New and replacement writes of an attested Connection require HTTPS;
  stored documents over plain HTTP stay readable and refused.

The JWKS cache race left by #568 is fixed first: a build now records a
save epoch before reading configuration or keys and is cached, with its
key problems, only if no save (local, or a reconcile that changes an
attested record) happened meanwhile. A fetch whose build was overtaken
builds again, and answers 503 if saves keep overtaking it, rather than
caching the pre-save set for a refresh interval.

Docs, schemas, the admin UI note and the CHANGELOG describe the real
behaviour, with a verifier example and an upgrade note: previously
refused attested Connections start serving manual tools once valid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant