Configurable way to allow entry/federation-relationship creation when a federated bundle is missing (revisiting #1256 / #4445)

[SpectralHiss]

Summary

When a registration entry's federatesWith references a trust domain with no
corresponding bundles row, SPIRE Server rejects the entry create/update
entirely:

unable to find federated bundle "spiffe://some-trust-domain"

This is enforced in makeFederatesWith
(pkg/server/datastore/sqlstore/sqlstore.go), which requires every trust
domain listed in federatesWith to resolve to an existing bundles row
before the entry write succeeds.

Real-world impact

We run federation from several per-cluster trust domains into a central
cluster, managed by an operator reconciling ClusterFederatedTrustDomain
resources. During an incident, one federated cluster's bundle failed to
publish to its storage backend (S3 404 / NoSuchKey). SPIRE Server had a
FederatedTrustDomain relationship for that trust domain registered, but no
bundles row. Our default ClusterSPIFFEID's federatesWith included that
trust domain among several others, so every entry generated from that CSID
failed to create — surfacing to workloads as PermissionDenied: no identity issued — even though only one of several federated trust domains was
actually unhealthy.

(To be clear: this isn't "the whole batch call fails" — BatchCreateEntry/
BatchUpdateEntry already process and report per-entry status correctly. The
blast radius here is topological: one broad federatesWith fans out to many
entries, and each independently hits the same existence check.)

History

• Creating entries with a FederatesWith for an unknown trust domain should succeed #1256 (2019): proposed the same fix. Discussion converged on a "sentinel
  bundle" design (@azdagron's
  comment):
  auto-create an empty bundle row (trust domain set, data column NULL) as
  a placeholder so the federated_registration_entries many-to-many join
  always resolves, with fetch/list operations filtering out NULL-data rows
  so placeholders are never mistaken for real bundles. This was chosen
  specifically to avoid the schema/data migration a "proper" fix would
  otherwise need (a decoupled table mapping entry ID -> trust domain
  strings, discussed
  here).
  @dfeldman implemented an auto-create-on-missing-bundle approach in If a bundle doesn't exist yet when creating a federation reg. entry, create it #1448
  (dfeldman:empty-bundle-federation). Note that this always created a real
  bundles row (never left a literal dangling foreign key — every entry's
  federatesWith still pointed at something that existed), but left two
  things unresolved by his own admission: placeholder bundles showed up in
  bundle list operations indistinguishable from real ones, and nothing
  cleaned them up when the owning relationship was removed.

  The issue and PR were ultimately closed won't-fix, but not over literal
  referential integrity (that was never actually at risk in If a bundle doesn't exist yet when creating a federation reg. entry, create it #1448's
  implementation) — @evan2645's closing rationale points back to the
  original 2019 intent behind the strict check: "we did this consciously
  with the intention of signaling to the registrant that something is up."
  Auto-creating a bundle on any federatesWith value removes that
  typo/misconfiguration signal, so a mistyped or stale trust domain string
  would succeed silently and leave a permanent, unintentional bundle row
  behind, with nothing to prompt an operator to notice. That's a real,
  distinct cost from FK-integrity risk, and it's the main thing an opt-in
  (rather than default-on) design needs to weigh.

• spire-oidc does not start if federatesWith contains a down federated endpoint  #4445 (2023): same failure recurred via spire-controller-manager /
  ClusterSPIFFEID. @evan2645 called it "pretty tough to fix" and asked for
  a volunteer; none appeared, closed without a fix.

Proposal

Make this opt-in and server-wide (no per-relationship granularity, no CRD
or spire-api-sdk changes, no schema migration), addressing the concerns
that sank #1256/#1448:

1. Sentinel bundle, never a dropped reference. When enabled via a
   server-wide config flag, and an entry references a trust domain with no
   bundles row, create a placeholder row (trust_domain set, data left
   NULL) instead of failing, inside makeFederatesWith. The entry keeps
   its federatesWith reference — nothing is silently dropped — so anything
   validating a peer against that trust domain's bundle still fails closed
   for that one relationship (zero authorities to validate against),
   instead of the workload getting no identity at all. No entry ever
   references a nonexistent row, only a temporarily-empty one, preserving
   the invariant Creating entries with a FederatesWith for an unknown trust domain should succeed #1256 was protecting.
2. NULL data is already legal today — no schema change needed.
   Bundle.Data (pkg/server/datastore/sqlstore/models.go) has no not null constraint. A real bundle — even one that happens to have zero
   authorities, which is a legitimate state a federated endpoint can publish
   ({} / {"keys":[]} is valid per pkg/common/bundleutil/unmarshal.go)
   — always has non-NULL data, because bundleToModel
   (sqlstore.go) always marshals at least TrustDomainId before writing
   it. So data IS NULL cleanly and unambiguously distinguishes "never
   resolved placeholder" from "resolved, happens to be empty" — a
   distinction decoded-authority-count alone cannot make.
3. Filter placeholders from FetchBundle/ListBundles, per
   @azdagron's original proposal, so they don't show up in spire-server bundle list/bundle show (or via GetFederatedBundle/
   ListFederatedBundles) as if they were real (if empty) published
   bundles.
4. No new cleanup mechanism on relationship deletion. Placeholder
   bundles are ordinary bundles rows. Deleting a FederationRelationship
   doesn't touch bundles today either, and shouldn't start now —
   spire-server bundle delete already defaults to RESTRICT, so any real
   entry dependency blocks accidental removal. Building bespoke cleanup here
   would risk recreating the exact dangling-reference bug this proposal
   avoids.
5. Operator UX parity. Losing the hard failure removes the original
   typo/misconfiguration signal (see "History"). Getting equivalent
   visibility back isn't a single change: today the error reaches CLI/API
   callers as an immediate, hard failure, and reaches spire-controller-manager
   as both a specific log line and a bump to the existing
   ClusterSPIFFEID status stats. Matching that means work in a few
   places: SPIRE Server surfacing a non-error warning on the affected
   create/update result, the spire-server CLI actually displaying it
   (today it only reads the result message on failure, not success), and
   spire-controller-manager surfacing the same signal somewhere operators
   already look (e.g. ClusterSPIFFEID/ClusterFederatedTrustDomain
   status). None of this comes for free just by adding the server-side
   flag.

Open questions for maintainers

• Trigger point: proactively in CreateFederationRelationship, or lazily in
  makeFederatesWith only when actually referenced by an entry? Leaning
  lazy, to avoid creating rows for trust domains nobody references yet.

Happy to put together a WIP PR if this direction seems reasonable to
maintainers.