Skip to content

Resolve caller-assertion signing keys and publish them as JWKS - #568

Merged
Antnation merged 7 commits into
mainfrom
feat/463-assertion-keys-jwks
Oct 20, 2026
Merged

Antnation merged 7 commits into
mainfrom
feat/463-assertion-keys-jwks

Conversation

@Antnation

@Antnation Antnation commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Part of #463

This resolves caller-assertion signing keys and publishes their public halves as a JWKS. It does not sign or inject assertions. The #556 refusal still applies, so every Connection that carries a caller_assertion block is refused on every lane with caller_assertion_not_enforced.

What changes

Keys. signing_key_id, and the new published_keys list of up to four next or previous keys, must each name a secret alias that holds exactly one PKCS#8 PEM private key. That means P-256 for es256, or RSA of at least 2048 bits for rs256.

  • Keys can come from operator aliases or external providers. The encrypted local store refuses this purpose.
  • Saving an enabled Connection loads every key. A key that cannot be used refuses the save with 422, the field, and a fixed code: unresolvable_binding, invalid_signing_key, algorithm_mismatch, unusable_signing_key or duplicate_kid.
  • A caller assertion counts as credential authority. Creating or deleting a Connection that carries one, or changing or removing the block, needs admin:connections:secrets:write. A routine save that keeps the block as stored does not.

JWKS. It is served at GET /.well-known/caller-assertion-jwks.json.

  • It is always public and exempt from authentication and RBAC. It is never proxied, and it carries public members only.
  • Responses carry Cache-Control: public, max-age=300 and a strong ETag, and a matching If-None-Match gets 304.
  • The set is rebuilt when the configured keys change, and otherwise every 300 seconds. A set that left a Connection out is rebuilt every 30 seconds.

Stored documents (maintainer decision). A stored Connection whose keys cannot be used no longer stops startup or fails a cluster reconcile.

  • Startup and reconcile check its keys apart from its credentials. A credential this replica cannot resolve still refuses the load, as before.
  • A Connection publishes all of its keys or none. If one of its keys cannot be used, all of its keys are left out of the JWKS. Every other Connection's keys are still published.
  • The problem is logged with the Connection, field and code. It also appears as configuration.caller_assertion.key_problem on the admin detail and as "Keys not published" in the admin UI.
  • Saving such a Connection with the block enabled is still refused with the codes above until the key is fixed. Disabling it needs no key.
  • The JWKS build uses the same per-Connection rule, including for a kid that two Connections publish for different keys. Both Connections are left out.

JWKS cache.

  • Builds no longer hold the cache lock while they read secrets.
  • A fetch that arrives during a rebuild for the same keys gets the previous set straight away.
  • A fetch with nothing it may serve waits at most 10 seconds, then gets 503 with no-store.
  • The snapshot is read again once the build lock is held, so a waiter cannot republish a withdrawn key.
  • Each alias is read once per build.
  • After a failed refresh, the gateway does not serve the keys it read earlier. What an unpinned alias holds may have changed, and the gateway does not use such an alias's material as a stale fallback. The operator guide records this choice.

Hardening.

  • The PEM check counts every BEGIN marker, because the parser skips blocks whose label it does not know.
  • The zeroization comment no longer overclaims.
  • The kid-uniqueness comment now says the save-time check is per replica, and that the JWKS build is the backstop.

Admin UI. The read-only caller-assertion view now shows published_keys and key_problem, so it matches the safe read and the published schema.

Docs.

  • docs/connections/operator-guide.md: key format and where keys may live, save-time codes, the JWKS contract and its trust scope (verifiers must pin iss and aud), degradation and diagnostics, a step-by-step rotation procedure, and an upgrade note.
  • docs/configuration.md: the JWKS path is exempt from authentication and RBAC whatever AUTH_EXEMPT_PATHS and RBAC_EXEMPT_PATHS say.
  • Both schema descriptions are updated, and CHANGELOG has a Security entry.

Merge with main. Main's #566 test now serves a generated PKCS#8 key through the counting provider. Setup goes through the same deferred-binding pass the admin API runs. The test asserts that the save read each secret exactly once, and that the refused call and the composite leaf read nothing more.

Transport review. Only gateway/src/app_state.rs:gateway::app_state::AppState#1 changed. Its sha256 comes from this head's inventory, and its purpose now records the review: the rename only exposes the existing control plane to the public JWKS handler, and an unauthenticated GET can trigger existing secret-provider egress, bounded by the JWKS cache.

Upgrade note

Earlier releases stored the block without reading its key. After upgrading, such a Connection still starts and still reconciles. It publishes no keys, reports key_problem, and stays refused. Its next save that keeps it enabled is refused until a usable key is behind its aliases, and that includes an admin UI save of another field. Fix the key, disable the Connection, or remove the block.

Tests

  • Write-time rejection: wrong algorithm, a published-key mismatch, duplicate_kid, and re-saving a degraded stored document.
  • Stored-document degradation:
    • restart with a corrupted key and with an unconfigured alias;
    • reconcile with an alias this replica lacks, while an unresolvable credential still fails the reconcile;
    • a key lost and recovered after a build;
    • a kid clash.
  • Runtime refusal on every degraded Connection.
  • An end-to-end router test showing the gateway starts and the JWKS publishes only the healthy key.
  • A gated-provider test showing that a fetch during a rebuild is not held, and that each alias is read once.
  • PEM tests for unrecognised blocks beside the key.
  • Admin UI projection and rendering tests.

Work in progress, parked when the implementing session stopped. Not yet compiled by CI or reviewed. Part of #463.
Main's #566 rewrote the caller-assertion refusal tests to serve the API
key and signing key from a counting provider, which proves the #556
refusal never consults a secret provider. That proof is kept. The
signing key it serves is now a generated PKCS#8 key, and the fixture
saves the Connection through the same deferred-binding pass the admin
API runs, so the key is really loaded. The tests assert that the save
read each secret exactly once, and that the refused call and the
composite leaf read nothing more.

Both schemas take main's opening for the caller_assertion block and
describe the keys this branch now resolves and publishes.
Releases since #556 stored an enabled Connection's caller_assertion
without reading its signing key, so a stored document can name an alias
that is unconfigured, a placeholder, or not a PKCS#8 key. Checking those
keys at load made ConnectionControlPlane::from_config refuse to start and
made a cluster reconcile fail for the whole snapshot, which takes every
other Connection down with one document.

Keys are now checked apart from credentials when a stored document is
loaded. Startup and reconcile still refuse a credential this replica
cannot resolve, as before, but a caller-assertion key that cannot be used
leaves only that Connection's keys out of the JWKS. It is logged with the
Connection, field and code, and reported as
configuration.caller_assertion.key_problem on the admin detail. The #556
refusal is unchanged, so the Connection stays refused on every lane. A
save is still held to the write-time rule and refused with the existing
safe codes until the key is usable; disabling needs no key.

The JWKS build follows the same rule: a Connection publishes all of its
keys or none. One whose key cannot be read or loaded, or that publishes a
kid another Connection publishes for a different key, is left out, and
the rest of the set is still published rather than the whole set going
503. A set that left a Connection out is rebuilt after 30 seconds instead
of 300.

The build no longer holds the cache lock across secret resolution. The
cache sits behind a short std lock, a separate async lock serializes
builds, and a fetch that arrives during a rebuild for the same keys is
answered from the previous set instead of queueing on the provider. A
fetch with nothing it may serve waits at most 10 seconds and then gets
503 no-store. The snapshot is read again once the build lock is held, so
a waiter can no longer rebuild from an older snapshot and republish a
withdrawn key. Each alias is resolved once per build. After a failed
refresh the gateway does not keep serving the last good keys: what an
unpinned alias holds may have changed, and the gateway does not use such
an alias's material as a stale fallback.

The PEM check now counts every BEGIN marker in the material, because the
parser skips blocks whose label it does not know. The kid-uniqueness
comment now says the write check is per replica and the JWKS is the
backstop.

The admin-router caller-assertion tests from #566 now bind a real key
and create with secrets-write, since a caller assertion is credential
authority. They assert that a routine rename by an editor still passes
and that removing the block without secrets-write is refused.
The safe read now carries a caller assertion's published_keys and, when
the gateway leaves the Connection's keys out of its JWKS, a key_problem
naming the field and a fixed code. The admin UI dropped both: its
projection kept only the fields it knew, so the editor contract test
that compares the fixture with the published schema would fail, and an
operator had no sign why a Connection's keys were missing.

The projection now validates and keeps both fields in the gateway's
serialization order. The read-only summary lists each published key and
shows an alert explaining the problem and how to clear it. It also says
that removing the block through the admin API needs secrets-write.
Issue #463 asks for the JWKS endpoint and its rotation procedure to be
documented. The operator guide said only that the block is stored, and
nothing about what this branch adds.

The Signed caller assertions section now covers the PKCS#8 and algorithm
requirements, where a key may live (never the encrypted local store),
the save-time error codes, and the secrets-write requirement. It
describes the public JWKS: authentication and RBAC exempt, never proxied,
public members only, one set for the whole gateway so verifiers must pin
iss and aud, caching and refresh, and the choice not to serve old keys
after a failed refresh. It explains the per-Connection degradation for
stored documents with its log line and key_problem, and gives a
step-by-step rotation procedure. An upgrade note says what changes for a
Connection stored by an earlier release. configuration.md records that
the JWKS path is exempt from authentication and RBAC independently of
AUTH_EXEMPT_PATHS and RBAC_EXEMPT_PATHS.
Renaming _connections to connections changes the syntax hash of the
reviewed AppState scope, so egress-only asked for a new review. The
review found no new transport authority. The struct only gains a name
and a doc comment, and its network reason comes from members this record
already covers. The rename exposes the existing control plane to the
public JWKS handler, so an unauthenticated GET can trigger existing
secret-provider reads. Those reads go only to operator-configured
destinations under the unchanged egress policy and the providers' own
deadlines, and the JWKS cache, one read per alias per build and the IP
rate-limit lane bound them.

Only this record changes: its hash comes from the guard's inventory of
this head, and its purpose now states that review.
@Antnation
Antnation marked this pull request as ready for review October 20, 2026 16:12
Copilot AI balanced review requested due to automatic review settings October 20, 2026 16:12
@Antnation Antnation changed the title WIP: Resolve caller-assertion signing keys and publish them as JWKS Resolve caller-assertion signing keys and publish them as JWKS Oct 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T16:16:14.913200Z 6065b8b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6065b8b370

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

|| path
.strip_prefix(WELL_KNOWN_PATH)
.is_some_and(|suffix| suffix.starts_with('/'))
|| path == crate::connections::caller_assertion::JWKS_PATH

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Run the cluster revision gate before serving JWKS

In cluster mode, do not include the JWKS in this shared bypass: rbac_middleware returns immediately for every is_well_known_path before invoking revision_gate.admit() (middleware/rbac.rs lines 1233-1261). If a replica is behind or fails Connection reconciliation after a key rotation or removal, it can therefore serve a stale JWKS that verifiers cache for 300 seconds; in particular, a retired or compromised key remains trusted even though normal protected requests fail closed. Keep the authentication/RBAC exemption, but ensure the JWKS still passes through the security-revision gate.

AGENTS.md reference: AGENTS.md:L53-L53

Useful? React with 👍 / 👎.

@Antnation Antnation mentioned this pull request Oct 20, 2026
@Antnation
Antnation merged commit 3f7723c into main Oct 20, 2026
35 checks passed
Antnation added a commit that referenced this pull request Oct 20, 2026
@Antnation
Antnation deleted the feat/463-assertion-keys-jwks branch October 20, 2026 16:41
Antnation added a commit that referenced this pull request Oct 21, 2026
Connections carrying a caller_assertion block have been refused on every
lane since #556 because nothing signed or injected the assertion. A manual
HTTP tool call is the lane that has both a validated principal and the
exact request in hand before it is sent, so it now attests its caller:

- ToolInvocationContext carries an AttestedCaller built from the
  authenticated auth::Principal where the call enters (MCP endpoint, admin
  playground), never from the audit actor. Service tokens and principals
  without an issuer yield none.
- ConnectionHttpRuntime::tool_call_target is the only constructor that
  serves an attested Connection, and only when the stored block passes the
  rules a new write is held to (bind_approval false, no header collision,
  HTTPS). Every other constructor keeps refusing with
  caller_assertion_not_enforced, so proxied requests, managed MCP,
  discovery, enum sources, stored tests, OpenAPI and composite tools are
  unchanged.
- A call with no principal is refused with caller_assertion_no_principal
  before the egress preflight, any secret read or any upstream byte; a
  Connection whose kid another enabled Connection publishes for a different
  key is refused with caller_assertion_key_not_published.
- Per call the signing key is resolved and loaded through the same path as
  the save and the JWKS, and one assertion is signed after every other
  header and the body are final: jti is the request id, iat/exp follow
  lifetime_seconds, and bind_request covers the method, the origin-form
  request target and the SHA-256 of the exact body bytes. It is written as
  the last header and refuses rather than replaces an existing value.
- Refusals use distinct fixed reasons through the #444 failure_reason
  path; key read failures also raise connection.secret_resolution_failed.
  tool.upstream_request records the assertion's kid and jti, never the
  token or key material.
- New and replacement writes of an attested Connection require HTTPS;
  stored documents over plain HTTP stay readable and refused.

The JWKS cache race left by #568 is fixed first: a build now records a
save epoch before reading configuration or keys and is cached, with its
key problems, only if no save (local, or a reconcile that changes an
attested record) happened meanwhile. A fetch whose build was overtaken
builds again, and answers 503 if saves keep overtaking it, rather than
caching the pre-save set for a refresh interval.

Docs, schemas, the admin UI note and the CHANGELOG describe the real
behaviour, with a verifier example and an upgrade note: previously
refused attested Connections start serving manual tools once valid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants