Repository navigation
Resolve caller-assertion signing keys and publish them as JWKS - #568
Conversation
Work in progress, parked when the implementing session stopped. Not yet compiled by CI or reviewed. Part of #463.
Main's #566 rewrote the caller-assertion refusal tests to serve the API key and signing key from a counting provider, which proves the #556 refusal never consults a secret provider. That proof is kept. The signing key it serves is now a generated PKCS#8 key, and the fixture saves the Connection through the same deferred-binding pass the admin API runs, so the key is really loaded. The tests assert that the save read each secret exactly once, and that the refused call and the composite leaf read nothing more. Both schemas take main's opening for the caller_assertion block and describe the keys this branch now resolves and publishes.
Releases since #556 stored an enabled Connection's caller_assertion without reading its signing key, so a stored document can name an alias that is unconfigured, a placeholder, or not a PKCS#8 key. Checking those keys at load made ConnectionControlPlane::from_config refuse to start and made a cluster reconcile fail for the whole snapshot, which takes every other Connection down with one document. Keys are now checked apart from credentials when a stored document is loaded. Startup and reconcile still refuse a credential this replica cannot resolve, as before, but a caller-assertion key that cannot be used leaves only that Connection's keys out of the JWKS. It is logged with the Connection, field and code, and reported as configuration.caller_assertion.key_problem on the admin detail. The #556 refusal is unchanged, so the Connection stays refused on every lane. A save is still held to the write-time rule and refused with the existing safe codes until the key is usable; disabling needs no key. The JWKS build follows the same rule: a Connection publishes all of its keys or none. One whose key cannot be read or loaded, or that publishes a kid another Connection publishes for a different key, is left out, and the rest of the set is still published rather than the whole set going 503. A set that left a Connection out is rebuilt after 30 seconds instead of 300. The build no longer holds the cache lock across secret resolution. The cache sits behind a short std lock, a separate async lock serializes builds, and a fetch that arrives during a rebuild for the same keys is answered from the previous set instead of queueing on the provider. A fetch with nothing it may serve waits at most 10 seconds and then gets 503 no-store. The snapshot is read again once the build lock is held, so a waiter can no longer rebuild from an older snapshot and republish a withdrawn key. Each alias is resolved once per build. After a failed refresh the gateway does not keep serving the last good keys: what an unpinned alias holds may have changed, and the gateway does not use such an alias's material as a stale fallback. The PEM check now counts every BEGIN marker in the material, because the parser skips blocks whose label it does not know. The kid-uniqueness comment now says the write check is per replica and the JWKS is the backstop. The admin-router caller-assertion tests from #566 now bind a real key and create with secrets-write, since a caller assertion is credential authority. They assert that a routine rename by an editor still passes and that removing the block without secrets-write is refused.
The safe read now carries a caller assertion's published_keys and, when the gateway leaves the Connection's keys out of its JWKS, a key_problem naming the field and a fixed code. The admin UI dropped both: its projection kept only the fields it knew, so the editor contract test that compares the fixture with the published schema would fail, and an operator had no sign why a Connection's keys were missing. The projection now validates and keeps both fields in the gateway's serialization order. The read-only summary lists each published key and shows an alert explaining the problem and how to clear it. It also says that removing the block through the admin API needs secrets-write.
Issue #463 asks for the JWKS endpoint and its rotation procedure to be documented. The operator guide said only that the block is stored, and nothing about what this branch adds. The Signed caller assertions section now covers the PKCS#8 and algorithm requirements, where a key may live (never the encrypted local store), the save-time error codes, and the secrets-write requirement. It describes the public JWKS: authentication and RBAC exempt, never proxied, public members only, one set for the whole gateway so verifiers must pin iss and aud, caching and refresh, and the choice not to serve old keys after a failed refresh. It explains the per-Connection degradation for stored documents with its log line and key_problem, and gives a step-by-step rotation procedure. An upgrade note says what changes for a Connection stored by an earlier release. configuration.md records that the JWKS path is exempt from authentication and RBAC independently of AUTH_EXEMPT_PATHS and RBAC_EXEMPT_PATHS.
Renaming _connections to connections changes the syntax hash of the reviewed AppState scope, so egress-only asked for a new review. The review found no new transport authority. The struct only gains a name and a doc comment, and its network reason comes from members this record already covers. The rename exposes the existing control plane to the public JWKS handler, so an unauthenticated GET can trigger existing secret-provider reads. Those reads go only to operator-configured destinations under the unchanged egress policy and the providers' own deadlines, and the JWKS cache, one read per alias per build and the IP rate-limit lane bound them. Only this record changes: its hash comes from the guard's inventory of this head, and its purpose now states that review.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6065b8b370
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| || path | ||
| .strip_prefix(WELL_KNOWN_PATH) | ||
| .is_some_and(|suffix| suffix.starts_with('/')) | ||
| || path == crate::connections::caller_assertion::JWKS_PATH |
There was a problem hiding this comment.
Run the cluster revision gate before serving JWKS
In cluster mode, do not include the JWKS in this shared bypass: rbac_middleware returns immediately for every is_well_known_path before invoking revision_gate.admit() (middleware/rbac.rs lines 1233-1261). If a replica is behind or fails Connection reconciliation after a key rotation or removal, it can therefore serve a stale JWKS that verifiers cache for 300 seconds; in particular, a retired or compromised key remains trusted even though normal protected requests fail closed. Keep the authentication/RBAC exemption, but ensure the JWKS still passes through the security-revision gate.
AGENTS.md reference: AGENTS.md:L53-L53
Useful? React with 👍 / 👎.
Connections carrying a caller_assertion block have been refused on every lane since #556 because nothing signed or injected the assertion. A manual HTTP tool call is the lane that has both a validated principal and the exact request in hand before it is sent, so it now attests its caller: - ToolInvocationContext carries an AttestedCaller built from the authenticated auth::Principal where the call enters (MCP endpoint, admin playground), never from the audit actor. Service tokens and principals without an issuer yield none. - ConnectionHttpRuntime::tool_call_target is the only constructor that serves an attested Connection, and only when the stored block passes the rules a new write is held to (bind_approval false, no header collision, HTTPS). Every other constructor keeps refusing with caller_assertion_not_enforced, so proxied requests, managed MCP, discovery, enum sources, stored tests, OpenAPI and composite tools are unchanged. - A call with no principal is refused with caller_assertion_no_principal before the egress preflight, any secret read or any upstream byte; a Connection whose kid another enabled Connection publishes for a different key is refused with caller_assertion_key_not_published. - Per call the signing key is resolved and loaded through the same path as the save and the JWKS, and one assertion is signed after every other header and the body are final: jti is the request id, iat/exp follow lifetime_seconds, and bind_request covers the method, the origin-form request target and the SHA-256 of the exact body bytes. It is written as the last header and refuses rather than replaces an existing value. - Refusals use distinct fixed reasons through the #444 failure_reason path; key read failures also raise connection.secret_resolution_failed. tool.upstream_request records the assertion's kid and jti, never the token or key material. - New and replacement writes of an attested Connection require HTTPS; stored documents over plain HTTP stay readable and refused. The JWKS cache race left by #568 is fixed first: a build now records a save epoch before reading configuration or keys and is cached, with its key problems, only if no save (local, or a reconcile that changes an attested record) happened meanwhile. A fetch whose build was overtaken builds again, and answers 503 if saves keep overtaking it, rather than caching the pre-save set for a refresh interval. Docs, schemas, the admin UI note and the CHANGELOG describe the real behaviour, with a verifier example and an upgrade note: previously refused attested Connections start serving manual tools once valid.
Part of #463
This resolves caller-assertion signing keys and publishes their public halves as a JWKS. It does not sign or inject assertions. The #556 refusal still applies, so every Connection that carries a
caller_assertionblock is refused on every lane withcaller_assertion_not_enforced.What changes
Keys.
signing_key_id, and the newpublished_keyslist of up to four next or previous keys, must each name a secret alias that holds exactly one PKCS#8 PEM private key. That means P-256 fores256, or RSA of at least 2048 bits forrs256.422, the field, and a fixed code:unresolvable_binding,invalid_signing_key,algorithm_mismatch,unusable_signing_keyorduplicate_kid.admin:connections:secrets:write. A routine save that keeps the block as stored does not.JWKS. It is served at
GET /.well-known/caller-assertion-jwks.json.Cache-Control: public, max-age=300and a strongETag, and a matchingIf-None-Matchgets304.Stored documents (maintainer decision). A stored Connection whose keys cannot be used no longer stops startup or fails a cluster reconcile.
configuration.caller_assertion.key_problemon the admin detail and as "Keys not published" in the admin UI.kidthat two Connections publish for different keys. Both Connections are left out.JWKS cache.
503withno-store.Hardening.
BEGINmarker, because the parser skips blocks whose label it does not know.Admin UI. The read-only caller-assertion view now shows
published_keysandkey_problem, so it matches the safe read and the published schema.Docs.
docs/connections/operator-guide.md: key format and where keys may live, save-time codes, the JWKS contract and its trust scope (verifiers must pinissandaud), degradation and diagnostics, a step-by-step rotation procedure, and an upgrade note.docs/configuration.md: the JWKS path is exempt from authentication and RBAC whateverAUTH_EXEMPT_PATHSandRBAC_EXEMPT_PATHSsay.Merge with main. Main's #566 test now serves a generated PKCS#8 key through the counting provider. Setup goes through the same deferred-binding pass the admin API runs. The test asserts that the save read each secret exactly once, and that the refused call and the composite leaf read nothing more.
Transport review. Only
gateway/src/app_state.rs:gateway::app_state::AppState#1changed. Its sha256 comes from this head's inventory, and its purpose now records the review: the rename only exposes the existing control plane to the public JWKS handler, and an unauthenticated GET can trigger existing secret-provider egress, bounded by the JWKS cache.Upgrade note
Earlier releases stored the block without reading its key. After upgrading, such a Connection still starts and still reconciles. It publishes no keys, reports
key_problem, and stays refused. Its next save that keeps it enabled is refused until a usable key is behind its aliases, and that includes an admin UI save of another field. Fix the key, disable the Connection, or remove the block.Tests
duplicate_kid, and re-saving a degraded stored document.