Repository navigation
Refuse calls on Connections with an unenforced caller_assertion - #556
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 411dec743e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .managed() | ||
| .get(connection_id) | ||
| .ok_or(ConnectionHttpError::ConnectionNotFound)?; | ||
| refuse_unenforced_caller_assertion(record)?; |
There was a problem hiding this comment.
Filter refused OpenAPI catalogs before registry validation
When upgrading a database that already contains both a caller_assertion Connection and a nonempty registered OpenAPI catalog—both accepted by the previous release—the boot loader still classifies that catalog as active in OpenApiConnectionCatalogService::load_with_enum_sources. Its definitions then enter the registry validator, which calls validate_binding; this new refusal makes the merge return an error and aborts gateway startup. Treat these catalogs like disabled-Connection catalogs by excluding or withdrawing them before registry validation, while retaining the durable catalog for later reactivation.
Useful? React with 👍 / 👎.
| | ConnectionHttpError::ConnectionNotFound | ||
| | ConnectionHttpError::WrongConnectionKind | ||
| | ConnectionHttpError::UnsupportedAuthentication | ||
| | ConnectionHttpError::CallerAssertionNotEnforced |
There was a problem hiding this comment.
Audit caller-assertion proxy refusals
When a connection-bound proxy request hits this new branch, the refusal reason is emitted only through tracing::warn; the UpstreamOutcome attached to the response has no reason field, so the structured http.request_observed audit records only a generic 502 with zero attempts and cannot identify the caller-assertion security decision. Emit a dedicated audit event or propagate the bounded reason into the structured request audit.
AGENTS.md reference: AGENTS.md:L55-L55
Useful? React with 👍 / 👎.
The Connection model and admin API have accepted and persisted a caller_assertion block since it was added, but nothing on the request path mints or injects the assertion. A Connection carrying one was served exactly like one without it, so the upstream received calls attested by nobody, including calls that arrived with no principal at all. That is a silent fail-open for an operator who configured every call to name its caller. Refuse such a Connection where every lane gets its target. All five target constructors now read the record through one helper that rejects a caller_assertion with CallerAssertionNotEnforced, and a ConnectionHttpTarget is the only way tool calls, the proxy, managed MCP sessions and discovery, OpenAPI discovery, enum and label sources and Connection tests reach an egress client, TLS identity or credential. The refusal therefore lands before DNS, secret reads, OAuth minting and any upstream byte, and no lane can route around it. Placing it in credential resolution instead would have been too late: every lane runs the egress preflight and resolves TLS material before it resolves the credential. The refusal keeps the fixed reason caller_assertion_not_enforced on every surface: the tool work-failure audit (#444 failure_reason), the Connection test stage reason (also accepted by the admin UI), the OpenAPI and MCP catalog errors (409), enum source failures, and the proxy's error category (502). New and replacement writes also reject bind_approval true, since there is no approval subsystem a mutation could be bound to, and an assertion header_name equal to any additional header name in any case. Documents persisted before those rules stay readable through the v0 persisted reader, as legacy OPTIONS test profiles do, so an upgrade cannot turn the store unreadable; they are refused at runtime like every Connection that carries an assertion. The published schemas and the operator guide say the block is accepted but not yet enforced.
Only ES256 had a sign/verify test. RS256 needs an RSA key, and no private key belongs in this tree, but neither rcgen (built on ring here) nor jsonwebtoken can generate one. aws-lc-rs can, and it is already in the graph as jsonwebtoken's backend at the same 1.18.1, so it becomes a dev-dependency edge rather than a new crate; rcgen only wraps the generated PKCS#8 DER as PEM. The test signs with the generated key and verifies with a standard verifier holding nothing but the JWK this gateway publishes for it, which also covers rs256_public_jwk. A second generated key's JWK must refuse the token, and the RSA key must not load as an ES256 key. The transport-ownership graph records the new dev-dependency edge; no crate, feature or syntax scope changed.
The admin UI does not model caller_assertion, and the safe read replaces its signing key alias with signing_key_configured, so the UI cannot send the block back. The replace handler read the missing key as removal, so any routine save from the UI (a display name, the enabled flag, a secret rotation) stripped the block. That lifted the caller_assertion_not_enforced refusal and put the Connection back into service with no assertion: the same silent fail-open #463 is about. A replacement that omits the key now keeps the current block, the way omitted hidden secret bindings are already kept. Removing the block takes an explicit "caller_assertion": null, and the replace request schema now accepts that null. A Connection that never had the block is unaffected.
validate_binding now refuses a Connection with a caller_assertion, and the tool registry runs that binding validation over the merged definitions of every lane on every change. Without this change, a live managed OpenAPI catalog on such a Connection would break more than its own tools. After an admin added the block to a Connection with a registered catalog, every later registry change (a TOOLS_FILE reload, another Connection's OpenAPI or MCP refresh) would fail validation. A catalog registered on a Connection that already had the block, which register allowed before this branch, would stop the gateway from starting after an upgrade, because the boot merge would refuse it. Such a catalog is now withdrawn from the registry and kept in storage, as a disabled Connection's catalog already is: at boot, in the authority reconcile and when the Connection changes. Its tools would be refused on every call anyway. Tests cover the withdrawal against the real binding validator, and the refusal of preview, register and refresh with caller_assertion_not_enforced before any spec fetch.
Every lane was already refused at target construction, but several surfaces were only checked for "some error" or for a missing event, which would pass for the wrong reason too: - MCP catalog refresh mapped the target error by matching reason strings and fell back to discovery_not_configured on anything it did not know, so a renamed reason would have changed the result silently. It now matches the error variant. Tests cover refresh on an attested Connection and tie the session-call reason string to the typed error. - OpenAPI and MCP catalog refusals are asserted as 409 with caller_assertion_not_enforced. - The proxy test asserted only a 502 with no attempts, which any target error produces. It now captures the fail-closed log line, the only place the proxy records why, and checks it names the refusal and leaks no secret. - The executor test checked for no secret-resolution event while both secrets still resolved, so the event was absent either way. A second call now runs after both secret files are removed. A provider read before the refusal would then fail as credential_unavailable and raise that event. The reason stays caller_assertion_not_enforced and the event never appears. - A composite leaf on such a Connection is refused before member I/O. The invocation reports composite_failed, and the leaf reason reaches the caller's details.
The operator guide and CHANGELOG said every refusal carries the reason in the tool audit failure_reason. Two lanes do not. A composite tool's audit failure_reason is composite_failed, and the step's reason reaches only the error details returned to the caller. A proxied request records the reason only as error_category on the gateway's warning log line, as for every other Connection failure on a proxy route. Both now say so, lane by lane. They also now cover the upgrade impact: a TOOLS_FILE tool or proxy route bound to such a Connection fails startup or the reload that contains it, the same as a binding to a disabled Connection. They describe the OpenAPI catalog withdrawal and the PUT rule that keeps an omitted block. The schema descriptions lead with the fact that the block is not yet enforced and describe signing as what will happen, not as what happens today.
411dec7 to
13bf909
Compare
Main's #566 rewrote the caller-assertion refusal tests to serve the API key and signing key from a counting provider, which proves the #556 refusal never consults a secret provider. That proof is kept. The signing key it serves is now a generated PKCS#8 key, and the fixture saves the Connection through the same deferred-binding pass the admin API runs, so the key is really loaded. The tests assert that the save read each secret exactly once, and that the refused call and the composite leaf read nothing more. Both schemas take main's opening for the caller_assertion block and describe the keys this branch now resolves and publishes.
Releases since #556 stored an enabled Connection's caller_assertion without reading its signing key, so a stored document can name an alias that is unconfigured, a placeholder, or not a PKCS#8 key. Checking those keys at load made ConnectionControlPlane::from_config refuse to start and made a cluster reconcile fail for the whole snapshot, which takes every other Connection down with one document. Keys are now checked apart from credentials when a stored document is loaded. Startup and reconcile still refuse a credential this replica cannot resolve, as before, but a caller-assertion key that cannot be used leaves only that Connection's keys out of the JWKS. It is logged with the Connection, field and code, and reported as configuration.caller_assertion.key_problem on the admin detail. The #556 refusal is unchanged, so the Connection stays refused on every lane. A save is still held to the write-time rule and refused with the existing safe codes until the key is usable; disabling needs no key. The JWKS build follows the same rule: a Connection publishes all of its keys or none. One whose key cannot be read or loaded, or that publishes a kid another Connection publishes for a different key, is left out, and the rest of the set is still published rather than the whole set going 503. A set that left a Connection out is rebuilt after 30 seconds instead of 300. The build no longer holds the cache lock across secret resolution. The cache sits behind a short std lock, a separate async lock serializes builds, and a fetch that arrives during a rebuild for the same keys is answered from the previous set instead of queueing on the provider. A fetch with nothing it may serve waits at most 10 seconds and then gets 503 no-store. The snapshot is read again once the build lock is held, so a waiter can no longer rebuild from an older snapshot and republish a withdrawn key. Each alias is resolved once per build. After a failed refresh the gateway does not keep serving the last good keys: what an unpinned alias holds may have changed, and the gateway does not use such an alias's material as a stale fallback. The PEM check now counts every BEGIN marker in the material, because the parser skips blocks whose label it does not know. The kid-uniqueness comment now says the write check is per replica and the JWKS is the backstop. The admin-router caller-assertion tests from #566 now bind a real key and create with secrets-write, since a caller assertion is credential authority. They assert that a routine rename by an editor still passes and that removing the block without secrets-write is refused.
Connections carrying a caller_assertion block have been refused on every lane since #556 because nothing signed or injected the assertion. A manual HTTP tool call is the lane that has both a validated principal and the exact request in hand before it is sent, so it now attests its caller: - ToolInvocationContext carries an AttestedCaller built from the authenticated auth::Principal where the call enters (MCP endpoint, admin playground), never from the audit actor. Service tokens and principals without an issuer yield none. - ConnectionHttpRuntime::tool_call_target is the only constructor that serves an attested Connection, and only when the stored block passes the rules a new write is held to (bind_approval false, no header collision, HTTPS). Every other constructor keeps refusing with caller_assertion_not_enforced, so proxied requests, managed MCP, discovery, enum sources, stored tests, OpenAPI and composite tools are unchanged. - A call with no principal is refused with caller_assertion_no_principal before the egress preflight, any secret read or any upstream byte; a Connection whose kid another enabled Connection publishes for a different key is refused with caller_assertion_key_not_published. - Per call the signing key is resolved and loaded through the same path as the save and the JWKS, and one assertion is signed after every other header and the body are final: jti is the request id, iat/exp follow lifetime_seconds, and bind_request covers the method, the origin-form request target and the SHA-256 of the exact body bytes. It is written as the last header and refuses rather than replaces an existing value. - Refusals use distinct fixed reasons through the #444 failure_reason path; key read failures also raise connection.secret_resolution_failed. tool.upstream_request records the assertion's kid and jti, never the token or key material. - New and replacement writes of an attested Connection require HTTPS; stored documents over plain HTTP stay readable and refused. The JWKS cache race left by #568 is fixed first: a build now records a save epoch before reading configuration or keys and is cached, with its key problems, only if no save (local, or a reconcile that changes an attested record) happened meanwhile. A fetch whose build was overtaken builds again, and answers 503 if saves keep overtaking it, rather than caching the pre-save set for a refresh interval. Docs, schemas, the admin UI note and the CHANGELOG describe the real behaviour, with a verifier example and an upgrade note: previously refused attested Connections start serving manual tools once valid.
Part of #463
Problem
Since #480 the Connection schemas and admin API accept and persist a
caller_assertionblock, but nothing outside its own module reads it. A Connection that carries one is served exactly like one that does not: the upstream gets no assertion header, and calls that arrive without a principal are not refused. For an operator who configured "every call names its caller" that is a silent fail-open.What this changes
One refusal, before anything else happens. All five target constructors in
ConnectionHttpRuntime(target,mcp_target,openapi_discovery_target,test_target,mcp_test_target) now read their record through a singlemanaged_recordhelper that returnsConnectionHttpError::CallerAssertionNotEnforcedwhencaller_assertionis set. AConnectionHttpTargetis the only way any lane reaches an egress client, a TLS identity or a credential, so this covers:validate_binding) and proxy route construction, which reject a binding to such a Connection just as they reject a binding to a disabled oneI put the check in target construction rather than in
resolve_credential/resolve_credential_before. Every lane runs the egress preflight (DNS) andprepare_transport(TLS secret reads) before it resolves the credential. A check inside credential resolution would therefore come after egress-adjacent work, and it would not cover lanes that skip credentials (use_connection_authentication: false). The refusal at target construction comes before DNS, before any secret read, before OAuth minting and before any upstream byte.One fixed reason:
caller_assertion_not_enforced. Where it shows up:tool.invoke_failurewithreason: work_error,failure_reason: caller_assertion_not_enforced(#444 path);tool.upstream_requestfailure carries the same reasonfailure_reason: composite_failedin the audit event; the step'scaller_assertion_not_enforcedis in the error details returned to the caller only (unchanged composite behaviour)ConnectionTestReason::CallerAssertionNotEnforcedon theauthenticatedstage, statusrequest_failed(HTTP and MCP probe identical); the admin UI accepts the new reasonOpenApiCatalogError/McpCatalogRefreshErrorvariants, HTTP 409 like a disabled ConnectionEnumSourceFailureReason::CallerAssertionNotEnforcedinconnection.enum_source_refresh_failed(it used to fall through toegress_denied)bad_gateway; the reason is onlyerror_categoryon the existing fail-closed warning log line, not a structured audit event, as for every other Connection failure on a proxy routeNo key material is read, so none can leak. The executor test writes canary values behind both the API-key alias and the assertion signing-key alias, checks that neither shows up in the caller error, any audit event or the proxy log, and then repeats the call with both secret files removed. A provider read before the refusal would then fail as
credential_unavailableand raiseconnection.secret_resolution_failed. The reason stayscaller_assertion_not_enforcedand no such event appears.Keeping the refusal in place.
caller_assertion, and the safe read hides its signing key alias, so the UI cannot send the block back. The replace handler read the missing key as removal, so a routine UI save (display name, enabled flag, secret rotation) stripped the block and put the Connection back into service unattested. APUTthat omitscaller_assertionnow keeps the current block, as omitted hidden secret bindings already are. An explicit"caller_assertion": nullremoves it, and the replace request schema accepts thatnull.Validation (
validate_caller_assertion).bind_approval: trueis rejected withcaller_assertion.bind_approval/unsupported_binding. No approval subsystem exists, so the setting could only ever be unsatisfiable or ignored. The published request schemas pin it toconst: false.caller_assertion.header_namemust differ, case-insensitively, from everyadditional_headers[].header_name(duplicate_header_name). The existing primary-credential check is unchanged.validated_persisted_v0keeps documents written before these rules readable, the same way it handles the legacyOPTIONStest profile. Without that, one such row would make the store list fail and could stop the gateway from starting after an upgrade. Those documents are still refused at runtime, like every Connection with an assertion.RS256 round trip.
caller_assertion_tests.rsgenerates an RSA-2048 key at test runtime, signs with it, and verifies with jsonwebtoken using only the JWK thatrs256_public_jwkpublishes. It also checks that another key's JWK refuses the token and that the RSA key will not load as ES256. No PEM is committed. Generating the key needsaws-lc-rsas a dev-dependency. It is already in the graph as jsonwebtoken's backend at the same 1.18.1, so this adds a dev-only dependency edge ongateway, not a new crate, and no package's features change. rcgen is built on ring here and cannot generate RSA keys, and jsonwebtoken has no key generation.Docs. The operator guide's "Signed caller assertions" section says the block is accepted but not yet enforced, that Connections carrying it are refused until request-path injection ships, where the reason appears for each lane, the binding and catalog consequences, and how to remove the block. Both schemas' descriptions lead with that status, and the CHANGELOG Security entry carries an upgrade note.
Upgrade note. A
TOOLS_FILEtool or proxy route bound to a Connection that already carries the block (allowed since #480) now fails startup or the reload that contains it, not just that binding, the same as a binding to a disabled Connection.Rebase
Rebased onto
mainat 5d99076 (#549: Rust 1.99.0 / Node 26.10.0, jsonwebtoken 11.1.0, jsonschema 0.56, rustls 0.23.46). The CHANGELOG conflict keeps both Security entries.transport-ownership.jsonwas checked against the merged lockfile: the guard's dependency-review stage (graphandbuild_inputsequality withcargo metadata --locked --all-features) passes. Compared withmain, the only graph change is the dev-onlyaws-lc-rs@1.18.1edge ongateway, and scopes and build inputs are identical. The syntax stage needs the gateway build and runs in CI.Not in this PR (remaining #463 acceptance)
Not here: minting and injecting the assertion on the request path, refusing principal-less calls once injection exists, the JWKS endpoint and its rotation procedure,
bind_approvalonce an approval subsystem exists, and admin UI editing of the block. When injection lands, it has to lift the refusal added here deliberately. It also needs to decide whether an assertion should require HTTPS:requires_tlsdoes not considercaller_assertiontoday. That is harmless while nothing is sent, but it matters once a bearer assertion is. Also left for follow-up:requires_secrets_write_to_replacedoes not treat a change tocaller_assertion.signing_key_idas credential-authority work.Testing
build.rspins Node exactly):connections::http: every HTTP and MCP target constructor refuses,ConnectionTestServicereports the reason with no client prepared, and removing the block restores the targettools::executor_tests: refused tool call with auditfailure_reason, no canary in errors or audit, no upstream request, and a repeat with the secret files removed proving the provider is never read; composite leaf refused before member I/O with the leaf reason in detailsproxy::forward: refused proxied request, fixed 502 outcome with no attempts,error_categoryin the captured log, no canariesconnections::openapi: preview, register and refresh refused before any fetch; a live catalog is withdrawn when its Connection gains the block, and the registry still validates with the gateway's binding validatorconnections::mcp: refresh refused before any session; session-call reason tied to the typed errorapp_tests: PUT keeps an omitted block and removes it onnull; catalog refusals are 409 with the reasontools::enum_source: refusal after policy and before egressconnections::testandtools::mcp_upstream: classification parity between HTTP and MCPconnections::model: both validation rules, schemaconst, persisted readabilityconnections::caller_assertion: the RS256 round tripcargo fmt --all --checkwith the CI toolchainpython scripts/transport_guard.py check: dependency review passes against the merged lockfile (syntax stage in CI)