conductor-production-runbook.md

Conductor Production Operator Runbook

This runbook walks the full production lifecycle of a Conductor fleet with shipped Pipelock CLIs and a documented bring-your-own-PKI recipe — no custom Go and no hand-rolled OpenSSL beyond the PKI choice you make. It is the operational counterpart to the local dev walkthrough (which proves the round trip in-process with conductor bootstrap) and the Kubernetes deployment guide (which covers the Helm topology). Read those first for context; this guide is the operator workflow around a real fleet.

Everything here is Enterprise-tier. Every server command verifies a license granting the fleet feature and fails closed before binding a listener or writing a file. A follower running pipelock run with conductor.enabled: true also requires a license granting the fleet feature and fails closed before binding a listener. See pipelock license.

Build note. Conductor exists only in an enterprise build (-tags enterprise). The commands below assume a pipelock binary built with that tag. On an unofficial/local enterprise build, also set PIPELOCK_LICENSE_PUBLIC_KEY so license verification can run.

Lifecycle at a glance

Stage	Command(s)	Status
1. Generate fleet keys	pipelock signing key generate --purpose …	Shipped
2. Build the trust roster	pipelock signing roster build	Shipped
3. Provision PKI (BYO)	cert-manager recipe (below)	Shipped (recipe)
4. Deploy Conductor + sink	pipelock conductor serve / pipelock fleet-sink + Helm	Shipped
5. Enroll followers	enrollment_token_path auto-enroll or pipelock conductor enroll	Shipped
6. Publish a policy	pipelock conductor publish	Shipped
7. Retarget a follower stream	pipelock conductor publish --stream-switch-*	Shipped
8. Kill / resume the fleet	pipelock conductor kill / resume	Shipped
9. Roll back a bad bundle	pipelock conductor rollback	Shipped
10. Recover follower bundle state	pipelock conductor follower reset-bundle-state	Shipped
10b. Recover follower replay state	pipelock conductor follower reset-replay-state	Shipped
11. Mint enrollment tokens	pipelock conductor enrollment-token mint	Shipped
12. Fleet status / followers	pipelock conductor fleet status / followers	Shipped
13. Remove / decommission followers	pipelock conductor follower remove	Shipped
14. Back up / restore Conductor state	pipelock conductor store backup / restore	Shipped
15. Query the audit sink	pipelock conductor audit query	Shipped
16. Rotate certs and keys	cert-manager + signing key generate	Shipped

Every stage uses a shipped CLI; the full lifecycle runs without hand-rolled Go or OpenSSL beyond the documented bring-your-own-PKI choice. For the on-wire request shapes behind each command, see the audit-sink design.

0. License gate

Provision the Enterprise license first; every later command needs it.

# Install the signed Enterprise license token where pipelock reads it at startup.
# The token is the positional argument; --path is where it is written.
pipelock license install "$PIPELOCK_LICENSE_KEY" --path /etc/pipelock/license/license.token

# Confirm it verifies and grants the fleet feature.
pipelock license status

pipelock license status prints the tier, the granted features (look for fleet), and the expiry. If the fleet feature is absent, the server commands below refuse to start.

1. Generate the fleet keys

Conductor has five signing-key roles plus a per-follower recorder key. Generate each with pipelock signing key generate, which writes a 0600 JSON keypair file and prints the canonical sha256 fingerprint you pin downstream.

mkdir -p /etc/pipelock/fleet-keys && chmod 700 /etc/pipelock/fleet-keys

# Deployment-local trust root — signs the roster that followers pin.
pipelock signing key generate --purpose roster-root \
  --out /etc/pipelock/fleet-keys/fleet-root.json --id fleet-root

# Policy-bundle signing — signs every published policy bundle.
pipelock signing key generate --purpose policy-bundle-signing \
  --out /etc/pipelock/fleet-keys/policy-signing.json --id policy-primary

# Remote-kill signing — each approver holds one distinct threshold key.
pipelock signing key generate --purpose remote-kill-signing \
  --out /etc/pipelock/fleet-keys/kill-approver-1.json --id kill-approver-1

pipelock signing key generate --purpose remote-kill-signing \
  --out /etc/pipelock/fleet-keys/kill-approver-2.json --id kill-approver-2

# Policy-bundle rollback — signs rollback authorizations. THRESHOLD key.
pipelock signing key generate --purpose policy-bundle-rollback \
  --out /etc/pipelock/fleet-keys/rollback-approver-1.json --id rollback-approver-1

Threshold keys. Rollback, remote-kill, and trust-root-rotation are threshold roles: an authorization should require M independent approvers, not one. Generate one key file per approver on separate operator machines and configure the Conductor control plane to require the fleet threshold. Never deploy a single rollback or kill key as a one-signer authority. (The keygen command prints a threshold: required advisory line for these purposes on a build that carries the conductor-keygen help update.)

Reserved purposes. trust-root-rotation and enrollment-token-signing are wire-stable purpose bindings, but no shipped operator workflow consumes them yet. You can generate them, but nothing reads them until the trust-root rotation and enrollment-token signing workflows ship.

Each follower also needs an Ed25519 recorder key to sign its evidence:

# Per-follower recorder key, written into the pipelock keystore.
pipelock keygen edge-01-recorder
# Public key path is printed; mount the private key into the follower pod and
# point flight_recorder.signing_key_path at it.

2. Build the signed trust roster

Followers pin a trust roster — a signed list of which public keys are trusted for which purpose — by its root fingerprint. Build it from the root key and the public halves of the signing keys:

pipelock signing roster build \
  --root /etc/pipelock/fleet-keys/fleet-root.json \
  --include id=policy-primary,key=/etc/pipelock/fleet-keys/policy-signing.json,purpose=policy-bundle-signing,role=publisher \
  --include id=kill-approver-1,key=/etc/pipelock/fleet-keys/kill-approver-1.json,purpose=remote-kill-signing,role=kill-approver \
  --include id=kill-approver-2,key=/etc/pipelock/fleet-keys/kill-approver-2.json,purpose=remote-kill-signing,role=kill-approver \
  --include id=rollback-approver-1,key=/etc/pipelock/fleet-keys/rollback-approver-1.json,purpose=policy-bundle-rollback,role=rollback-approver \
  --data-class internal \
  --out /etc/pipelock/fleet-keys/trust-roster.json

# Confirm the roster signature and capture the root fingerprint to pin.
pipelock signing roster verify \
  --path /etc/pipelock/fleet-keys/trust-roster.json \
  --root-fingerprint sha256:<root-fingerprint-from-step-1>

roster build accepts either the JSON keypair file or a public-key file for each --include; it reads only the public half. Distribute trust-roster.json plus the root fingerprint to every follower (conductor.trust_roster_path and conductor.trust_roster_root_fingerprint). The fingerprint pin is what makes a swapped roster fail closed.

3. Provision PKI (bring-your-own, cert-manager recipe)

Conductor's mTLS is rooted in deployment-provided PKI, not the binary. The fleet CA signs two kinds of certificate:

• the Conductor server cert (DNS + IP SANs the followers reach it on);
• a per-follower client cert whose SPIFFE URI SAN is the follower's fleet identity — Conductor derives org/fleet/instance/environment from that URI, never from a request field, so a follower cannot impersonate another.

The proven recipe uses cert-manager with a CA ClusterIssuer. Generate the CA keypair once with your PKI tooling, store it in a Secret in the cert-manager namespace, and reference it:

# Fleet CA issuer. Signs BOTH the server cert and every follower client cert.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: conductor-fleet-ca
spec:
  ca:
    secretName: conductor-fleet-ca   # CA keypair Secret (cert-manager namespace)

# Conductor server cert, auto-renewed. SANs cover the in-cluster Service DNS and
# every node IP a follower might reach the leader on (e.g. via NodePort).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: conductor-server
  namespace: pipelock-control
spec:
  secretName: conductor-server-tls   # tls.crt / tls.key
  duration: 2160h                     # 90d
  renewBefore: 360h                   # 15d — renews well before expiry
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - server auth
  dnsNames:
    - conductor.pipelock-control.svc.cluster.local
    - conductor.pipelock-control.svc
  ipAddresses:
    - 127.0.0.1
    # - <node IPs here if followers reach the leader off-cluster via NodePort>
  issuerRef:
    name: conductor-fleet-ca
    kind: ClusterIssuer
    group: cert-manager.io

# Per-follower client cert. The SPIFFE URI SAN IS the follower's fleet identity.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: follower-edge-01-client
  namespace: pipelock
spec:
  secretName: follower-edge-01-client   # tls.crt / tls.key
  duration: 2160h
  renewBefore: 360h
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - client auth
  uris:
    - spiffe://pipelock.local/orgs/org-acme/fleets/prod/instances/edge-01/environments/prod
  issuerRef:
    name: conductor-fleet-ca
    kind: ClusterIssuer
    group: cert-manager.io

Issue one client Certificate per follower with a distinct SPIFFE URI. The --follower-trust-domain on conductor serve (default pipelock.local) must match the trust domain in these URIs. cert-manager handles auto-renewal mid-flight: renewBefore rotates the cert before it expires and the workload picks up the new Secret without a restart.

Off-cluster followers. A follower running outside Kubernetes (a bare-metal host) reaches the leader via a NodePort Service with externalTrafficPolicy: Local; add the node IPs it dials to the server cert ipAddresses. Its client cert is issued the same way — a Certificate with its own SPIFFE URI — and mounted on the host.

Don't want cert-manager? Any PKI that produces a CA, a server cert with the right SANs, and per-follower client certs with SPIFFE URI SANs works. The binary verifies the chain and the SPIFFE identity; you own the CA keys and renewal.

4. Deploy Conductor and the audit sink

Run the control plane. Operator bearer tokens are read from files (never flags), so they never land in process listings or shell history.

pipelock conductor serve \
  --listen 0.0.0.0:8895 \
  --probe-listen 0.0.0.0:9092 \
  --storage-dir /var/lib/pipelock/conductor \
  --conductor-id conductor-prod \
  --follower-trust-domain pipelock.local \
  --tls-cert /etc/pipelock/conductor/tls/server/tls.crt \
  --tls-key /etc/pipelock/conductor/tls/server/tls.key \
  --client-ca /etc/pipelock/conductor/tls/client-ca/ca.crt \
  --publisher-token-file /etc/pipelock/conductor/tokens/publisher/token \
  --publisher-org org-acme \
  --auditor-token-file /etc/pipelock/conductor/tokens/auditor/token \
  --auditor-org org-acme \
  --admin-token-file /etc/pipelock/conductor/tokens/admin/token \
  --admin-org org-acme \
  --trusted-audit-key "id=edge-01-audit,org=org-acme,inline=$(jq -r .public /etc/pipelock/fleet-keys/edge-01-audit.json)" \
  --trusted-control-key "id=kill-approver-1,purpose=remote-kill-signing,inline=$(jq -r .public /etc/pipelock/fleet-keys/kill-approver-1.json)" \
  --trusted-control-key "id=kill-approver-2,purpose=remote-kill-signing,inline=$(jq -r .public /etc/pipelock/fleet-keys/kill-approver-2.json)" \
  --trusted-control-key "id=rollback-approver-1,purpose=policy-bundle-rollback,inline=$(jq -r .public /etc/pipelock/fleet-keys/rollback-approver-1.json)" \
  --remote-kill-max-validity 72h \
  --rollback-max-validity 24h

Trusted-key format (a current rough edge). --trusted-audit-key and --trusted-control-key take the public key either inline= (a raw 64-hex Ed25519 public key) or file= (a public-key file in the versioned pipelock-ed25519-public-v1 format). pipelock signing key generate writes the public half as the public hex field inside its JSON keypair, so the simplest shipped path is inline=$(jq -r .public <keypair>.json) as shown above. There is no shipped command that emits a versioned .pub file for these conductor purposes yet, so prefer inline=. (The trust roster --include key= path is different — it reads the JSON keypair file directly.)

On Kubernetes, the Helm chart renders this from values-enterprise-conductor.yaml:

helm install pipelock-conductor ./charts/pipelock \
  -f charts/pipelock/examples/values-enterprise-conductor.yaml

Optionally run the standalone audit sink (or co-locate it with the control plane):

pipelock fleet-sink \
  --listen 0.0.0.0:8894 \
  --storage-dir /var/lib/pipelock/fleet-sink \
  --trusted-audit-key 'id=edge-01-audit,org=org-acme,file=/etc/pipelock/fleet-sink/audit-keys/edge-01.pub' \
  --tls-cert /etc/pipelock/fleet-sink/tls/tls.crt \
  --tls-key /etc/pipelock/fleet-sink/tls/tls.key \
  --client-ca /etc/pipelock/fleet-sink/tls/client-ca/ca.crt

Health, readiness, and Prometheus metrics live on the separate plain-HTTP probe listener (--probe-listen), never on the mTLS follower API.

5. Enroll followers

A follower is an ordinary Pipelock proxy with a conductor: block. It authenticates with its client cert, whose SPIFFE URI SAN is its fleet identity (org/fleet/instance/environment). Enrollment is token-gated: the follower registers its audit-batch public key with the Conductor by presenting a single-use enrollment token whose scope must match its certificate identity. Mint the token first (step 11), then deliver it to the follower one of two ways:

• Auto-enroll on startup (recommended). Point conductor.enrollment_token_path at a file containing the token. On startup the follower reads the token and POSTs it with its audit key id and audit public key over mTLS; the Conductor records the follower and trusts its audit key. Enrollment is best-effort and never blocks enforcement: a rejected or failed enroll logs a warning and the follower keeps serving and polling. A marker file under bundle_cache_dir makes normal restarts idempotent. If the leader accepts the token but the local marker write fails, the next restart may retry the already-consumed token and log a warning; enforcement still continues.
• One-shot operator command. Run pipelock conductor enroll (see step 11) with the token, the follower's audit key, and the follower's mTLS material. Use this when the follower is provisioned out of band.

Until a follower enrolls it still enforces locally and polls for policy and remote-kill, but it does not appear in conductor fleet status and the audit sink cannot verify its evidence — the Conductor has no audit key registered for it. Enrollment is what turns both on.

For a direct follower deployment, mount each path exactly as configured:

# pipelock-enterprise-skip-id: conductor-production-follower
conductor:
  enabled: true
  conductor_url: https://conductor.pipelock-control.svc.cluster.local:8895
  org_id: org-acme
  fleet_id: prod
  instance_id: edge-01
  trust_roster_path: /etc/pipelock/trust-roster.json
  # EXAMPLE ONLY: replace this with the fingerprint from step 1; do not deploy this value.
  trust_roster_root_fingerprint: sha256:e7064a5e10402f71d33b8629c63ae81e2fdf528594577a8ffa5febf0fa48c9ee
  server_ca_file: /etc/pipelock/conductor-ca.pem
  client_cert_path: /etc/pipelock/follower.crt
  client_key_path: /etc/pipelock/follower.key
  bundle_cache_dir: /var/lib/pipelock/bundles
  durable_audit_queue_dir: /var/lib/pipelock/audit-queue
  # Optional: omit to keep the existing plaintext-compatible durable queue.
  # Set only after provisioning the keyring outside the queue directory.
  durable_audit_queue_keyring: /etc/pipelock/secrets/audit-queue-keyring.json
  # audit_signing_key_id and recorder_key_id default to instance_id when omitted; override only if the audit sink expects a different key id
  # audit_signing_key_id: edge-01-audit
  # recorder_key_id: edge-01-recorder
  enrollment_token_path: /etc/pipelock/conductor/enrollment-token  # single-use; auto-enroll on startup
  poll_interval: 30s
  honor_remote_kill_switch: true

flight_recorder:
  enabled: true
  sign_checkpoints: true
  signing_key_path: /etc/pipelock/recorder.key

On ordinary server shutdown, the follower closes its recorder and audit producer, then gives the pending batches up to five seconds for a final delivery attempt before it closes the durable audit queue. A batch the Conductor refuses moves to the queue's dead-letter directory and the drain continues with the next batch. A retryable failure stops the drain and leaves that batch pending. Either outcome logs final conductor audit delivery incomplete. Fleet entitlement revocation cancels delivery immediately. The drain does not make a batch admissible after an outage: the Conductor still refuses an audit envelope outside its 60-second freshness window, so a batch queued before a long outage can land in dead letters. Inspect the pending/, inflight/, and dead/ directories under durable_audit_queue_dir before deleting any record.

A follower must produce signed evidence to participate: config validation rejects conductor.enabled: true unless the flight recorder is enabled with signing checkpoints and a signing_key_path. The YAML above is a manual-mount example; do not copy its paths into Helm-generated configuration. On Kubernetes, use values-enterprise-follower.yaml:

Existing followers can upgrade with no queue-keyring change: when durable_audit_queue_keyring is unset, the follower keeps using the plaintext-compatible durable audit queue and emits a warning that queued records are unencrypted at rest.

To enable encryption, create a writable operator-owned source keyring outside the queue directory, then import it into a Kubernetes Secret mounted separately from the queue PVC. The chart renders durable_audit_queue_keyring from auditQueueKeyringSecretRef.mountPath and .key (the example resolves to /etc/pipelock/conductor/audit-queue-key/audit-queue-keyring.json). The mounted copy is runtime-only and read-only. Keep the operator source directory at 0700: it contains the raw symmetric key and has no group consumer.

install -d -m 700 "$PWD/operator-secrets"
pipelock conductor audit-queue-key init \
  --keyring "$PWD/operator-secrets/audit-queue-keyring.json"
kubectl -n pipelock create secret generic follower-audit-queue-keyring \
  --from-file=audit-queue-keyring.json="$PWD/operator-secrets/audit-queue-keyring.json"

For a live follower, scale down before providing the Secret and setting durable_audit_queue_keyring, then scale back up. On first start with the keyring configured, legacy plaintext queue records auto-encrypt under the active key before delivery resumes:

kubectl -n pipelock scale deployment pipelock-follower --replicas=0
kubectl -n pipelock create secret generic follower-audit-queue-keyring \
  --from-file=audit-queue-keyring.json="$PWD/operator-secrets/audit-queue-keyring.json"
# Update the follower config or Helm values to set durable_audit_queue_keyring
# to the mounted Secret path, then apply the rollout.
kubectl -n pipelock scale deployment pipelock-follower --replicas=1

Day-2 lifecycle commands require the follower to be stopped and must run in an operator maintenance environment that mounts the real queue PVC at /var/lib/pipelock/conductor/audit-queue while keeping the operator source keyring on a different writable volume. Inspection may use the read-only runtime mount; rotate, revoke, and recover must never target that Secret mount:

kubectl -n pipelock scale deployment pipelock-follower --replicas=0
export KEYRING="$PWD/operator-secrets/audit-queue-keyring.json"
export QUEUE_DIR=/var/lib/pipelock/conductor/audit-queue
pipelock conductor audit-queue-key inspect --keyring "$KEYRING" --queue-dir "$QUEUE_DIR"
pipelock conductor audit-queue-key rotate --keyring "$KEYRING" --queue-dir "$QUEUE_DIR"
pipelock conductor audit-queue-key migrate --keyring "$KEYRING" --queue-dir "$QUEUE_DIR"
pipelock conductor audit-queue-key revoke sha256:0123456789abcdef --keyring "$KEYRING" --queue-dir "$QUEUE_DIR"
pipelock conductor audit-queue-key recover --backup "$KEYRING.bak" --keyring "$KEYRING" --queue-dir "$QUEUE_DIR"
kubectl -n pipelock create secret generic follower-audit-queue-keyring \
  --from-file=audit-queue-keyring.json="$KEYRING" \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n pipelock scale deployment pipelock-follower --replicas=1

Rotation saves the pre-rotation keyring as KEYRING.bak.previous, then writes the rotated keyring—with both old and new decryptors—to the live file and KEYRING.bak before re-encrypting records. An interruption therefore retains both decryptors; rerun migrate to converge. Revocation refuses the active key and any key still referenced by a pending, inflight, or dead-letter record. Recovery first proves KEYRING.bak decrypts every queued record, preserves the current live keyring as KEYRING.pre-recover, then restores the backup verbatim. That intentionally reinstates any keys revoked after the backup; use the pre-recovery snapshot to reverse an accidental recovery. Use the inactive key ID printed by inspect in place of the example sha256:0123456789abcdef value.

Kubernetes Secret mounts are read-only, so retain the operator-owned source copy. Never rotate against an empty temporary queue: maintenance commands now require the initialized real queue layout and inspect every durable state before mutation. Update the Secret only after the command succeeds, then scale the Deployment back up. Startup verifies and migrates every PVC record before delivery resumes. After inspect reports zero records for the old ID, revoke it in the source file and repeat the scale-down/update/scale-up flow. Never delete the old key from the Secret before PVC migration has completed.

helm install pipelock-follower ./charts/pipelock \
  -f charts/pipelock/examples/values-enterprise-follower.yaml

Once running, the follower polls Conductor on poll_interval for the latest policy bundle and for remote-kill state, and ships signed audit batches to the sink. These three loops are what Conductor coordinates — the follower still enforces locally and stays fail-closed if Conductor is unreachable.

6. Publish a policy bundle

pipelock conductor publish builds, signs, and POSTs a policy bundle from a follower config. The bundle is signed with a policy-bundle-signing key and authorized by the publisher token; followers verify the signature against the pinned roster and apply only bundles addressed to their org/fleet/environment/ audience.

pipelock conductor publish \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --config /etc/pipelock/fleet-policy.yaml \
  --bundle-id prod-2026-06-11 \
  --org org-acme --fleet prod --env prod \
  --validity 720h \
  --min-pipelock-version 2.7.0 \
  --signing-key /etc/pipelock/fleet-keys/policy-signing.json \
  --publisher-token-file /etc/pipelock/conductor/tokens/publisher/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

The command computes the canonical policy_hash / payload_sha256, stamps the validity window from --validity, and carries the operator-supplied monotonic --version.

A follower that cannot prove its own released version (a dev or source build) always refuses a bundle carrying --min-pipelock-version and keeps its currently active policy. This refusal ignores rules.allow_unversioned_bundle_load, which governs only standalone rule-bundle loading (pipelock rules install) and defaults to warn-and-load there; Conductor's minimum-version check is stricter and has no equivalent setting. Add --rule-bundle <path> (repeatable) to ship signed rule bundles alongside the config, --audience to narrow the addressed followers, and --previous-bundle-hash <hex> to pin continuity against the prior bundle (the publish is rejected if the server's current bundle does not match). For a loopback dev conductor without TLS, --allow-plaintext-loopback permits an http://127.0.0.1 URL.

Add --dry-run to send the same signed bundle to Conductor's read-only publish evaluation endpoint. Conductor returns the publish evaluation, including whether the bundle would be created, the resulting hash/version, fleet preflight summary, and any conflict code, without mutating the bundle store. Older Conductors that do not expose the evaluation endpoint fail closed instead of silently treating a dry-run as a publish.

Prove on apply: after publishing, confirm followers picked up the new version (watch the follower logs for the bundle-apply line, or pipelock conductor fleet status) before treating the rollout as complete.

7. Retarget a follower stream

A follower can match more than one policy stream, such as the fleet wildcard audience (*) and its own instance audience. When moving a running follower to a more-specific stream, publish the target bundle with a stream-switch authorization. The authorization is signed by the policy-bundle-rollback threshold keys and binds the source bundle hash to the target bundle before the target bundle is policy-signed.

pipelock conductor publish \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --config /etc/pipelock/edge-01-policy.yaml \
  --bundle-id edge-01-canary-v11 \
  --org org-acme --fleet prod --env prod \
  --audience edge-01 \
  --version 11 \
  --validity 720h \
  --min-pipelock-version 2.9.0 \
  --signing-key /etc/pipelock/fleet-keys/policy-signing.json \
  --stream-switch-from-audience '*' \
  --stream-switch-from-bundle-id prod-wildcard-v10 \
  --stream-switch-from-version 10 \
  --stream-switch-from-hash 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --stream-switch-reason "move edge-01 to canary policy" \
  --stream-switch-signing-key /etc/pipelock/fleet-keys/rollback-approver-1.json \
  --stream-switch-signing-key /etc/pipelock/fleet-keys/rollback-approver-2.json \
  --publisher-token-file /etc/pipelock/conductor/tokens/publisher/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

Use pipelock conductor stream status to inspect the source stream head hash before publishing. A follower rejects a cross-stream target without the signed authorization, with an expired authorization, or when the authorization names a different source hash, target bundle, or audience.

8. Kill and resume the fleet

pipelock conductor kill pushes a signed, time-bounded fleet-wide kill; pipelock conductor resume clears it. Followers with honor_remote_kill_switch: true fail closed within one poll and recover on resume.

# Push a signed, time-bounded fleet-wide kill (threshold-signed).
pipelock conductor kill \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org org-acme --fleet prod --instance '*' \
  --signing-key /etc/pipelock/fleet-keys/kill-approver-1.json \
  --signing-key /etc/pipelock/fleet-keys/kill-approver-2.json \
  --ttl 1h \
  --admin-token-file /etc/pipelock/conductor/tokens/admin/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

# Clear it once the incident is resolved.
pipelock conductor resume \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org org-acme --fleet prod --instance '*' \
  --signing-key /etc/pipelock/fleet-keys/kill-approver-1.json \
  --signing-key /etc/pipelock/fleet-keys/kill-approver-2.json \
  --admin-token-file /etc/pipelock/conductor/tokens/admin/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

Conductor rejects a kill whose validity window exceeds --remote-kill-max-validity (default 72h). Followers honoring the kill switch fail closed within one poll_interval; clearing it lets them recover on the next poll.

Add --dry-run to kill or resume to sign the same remote-kill message and ask Conductor's read-only evaluation endpoint for the verdict. The response reports the would-be counter/hash decision and conflict code without publishing the kill/resume state.

9. Roll back a bad bundle

pipelock conductor rollback authorizes a signed revert to a prior bundle version for the policy stream selected by the current and target bundle pair. Other policy streams in the same organization and fleet remain unchanged. Per-instance and per-label rollback within that stream are not supported.

pipelock conductor rollback \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org org-acme --fleet prod \
  --current-bundle-id bundle-v2 --current-version 2 \
  --target-bundle-id bundle-v1 --target-version 1 \
  --signing-key /etc/pipelock/fleet-keys/rollback-approver-1.json \
  --signing-key /etc/pipelock/fleet-keys/rollback-approver-2.json \
  --ttl 1h \
  --admin-token-file /etc/pipelock/conductor/tokens/admin/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

Signed with the policy-bundle-rollback (threshold) key. Conductor rejects a rollback whose window exceeds --rollback-max-validity (default 24h). Because kill and rollback keys are purpose-scoped, a rollback key cannot issue a kill or vice versa.

Add --dry-run to sign the same rollback authorization and ask Conductor's read-only evaluation endpoint for the verdict. The response reports whether the authorization would be created and which bundle/version/hash the stream head would roll to, without writing either the authorization or stream-head mutation.

9a. Replay a publish decision

pipelock conductor replay --bundle-artifact <bundle.json> POSTs an exact previously saved signed policy-bundle artifact to Conductor's decision-replay endpoint. Replay re-derives the publish verdict under current store/fleet state and reports whether that diverges from any recorded accepted decision.

pipelock conductor replay \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --bundle-artifact /var/lib/pipelock/conductor/exported/prod-2026-06-11.json \
  --publisher-token-file /etc/pipelock/conductor/tokens/publisher/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

Without --bundle-artifact, replay rebuilds and signs a new hypothetical bundle from the same inputs as conductor publish. Use that mode for "what would this input do now?" checks, not for exact audit replay, because the rebuilt artifact has fresh timestamps and signatures.

Pass --state-snapshot snapshot.json to replay the publish preflight against a captured follower/runtime-status snapshot. The snapshot applies only to bundle replay preflight; the policy-bundle store chain is still evaluated against current Conductor state.

9b. Follower startup and consistency

On restart, a follower does not trust its cached last-known-good bundle blindly: it re-verifies the signature, audience, not_before, and local-version gates against that cached bundle before reloading it into the running proxy, the same checks a fresh apply runs. Admission is marked uncertain for the duration of that reload; if the reload fails, the follower denies admission and reports live policy as uncertain rather than serving traffic under an unconfirmed config. A failed activation during a normal (non-startup) apply reverts to the prior live policy when a restore path is available, rather than leaving the runtime and the durable "active" pointer disagreeing about which policy is live. See §10 if a follower gets stuck in this state and needs its local bundle cache reset.

10. Recover follower bundle state

If a follower's local policy-bundle apply cache is wedged, reset only that local bundle state on the follower host. This does not touch remote-kill replay state.

# Dry run first.
pipelock conductor follower reset-bundle-state \
  --state-dir /var/lib/pipelock/bundles

# Apply after confirming the state dir is conductor.bundle_cache_dir.
pipelock conductor follower reset-bundle-state \
  --state-dir /var/lib/pipelock/bundles \
  --confirm

Restart the follower after a confirmed reset if it is wedged. On the next poll it re-fetches and verifies the authoritative bundle for its audience.

10b. Recover follower replay state

If a follower is wedged with "conductor remote kill replay state missing while follower context is present," reset only the remote-kill replay state on the follower host. This does not touch policy-bundle apply state.

# Dry run first.
pipelock conductor follower reset-replay-state \
  --state-dir /var/lib/pipelock/bundles

# Apply after confirming the state dir is conductor.bundle_cache_dir.
pipelock conductor follower reset-replay-state \
  --state-dir /var/lib/pipelock/bundles \
  --confirm

Restart the follower after a confirmed reset. On the next poll it re-syncs the authoritative kill state from the Conductor.

11. Mint enrollment tokens

Enrollment is token-gated (see step 5). pipelock conductor enrollment-token mint issues a single-use token scoped to one follower identity (--org/--fleet/--instance/--env), authorized by an admin token over mTLS. --token-id is a stable id used for audit and de-duplication; --ttl (default 1h) bounds how long the token stays valid before it is used.

pipelock conductor enrollment-token mint \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org org-acme --fleet prod --instance edge-02 --env prod \
  --token-id edge-02-enroll-2026-06-11 --ttl 1h \
  --admin-token-file /etc/pipelock/conductor/tokens/admin/token \
  --tls-cert /etc/pipelock/operator.crt \
  --tls-key /etc/pipelock/operator.key \
  --server-ca /etc/pipelock/conductor-ca.pem

The command prints the token to stdout. For startup auto-enroll, write it to the follower's enrollment_token_path with restrictive permissions from the start (a umask in the same subshell so the file is never briefly world-readable):

( umask 077; pipelock conductor enrollment-token mint ... > /etc/pipelock/conductor/enrollment-token )

Or enroll an out-of-band follower directly with pipelock conductor enroll, which POSTs the token plus the follower's audit key id and audit public key over mTLS (--audit-key-file is the follower's audit-batch-signing key; a JSON keypair derives the key id, a raw private key needs --audit-key-id):

pipelock conductor enroll \
  --conductor-url https://conductor.pipelock-control.svc.cluster.local:8895 \
  --enrollment-token-file /etc/pipelock/conductor/enrollment-token \
  --audit-key-file /etc/pipelock/conductor/edge-02-audit.key \
  --audit-key-id edge-02-audit \
  --tls-cert /etc/pipelock/follower.crt \
  --tls-key /etc/pipelock/follower.key \
  --server-ca /etc/pipelock/conductor-ca.pem

Troubleshooting: re-enrolling after an aborted attempt

If enrollment returns a 409 ("conductor follower instance already enrolled"), the follower identity is still registered on the Conductor. To re-enroll:

1. Remove the existing enrollment on the server side:

pipelock conductor follower remove \
  --server https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org-id <org> --fleet-id <fleet> --instance-id <instance> --environment <env> \
  --token-file /etc/pipelock/conductor/tokens/admin/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

2. Mint a new enrollment token (enrollment-token mint).
3. Re-enroll the follower using the new token.

12. Fleet status and followers

pipelock conductor fleet status lists enrolled instances — identity, audit key id, active state, and enrollment time — read over mTLS with an org-scoped operator token (admin or auditor) against the audience-scoped /followers endpoint.

pipelock conductor fleet status \
  --server https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org-id org-acme --fleet-id prod \
  --token-file /etc/pipelock/conductor/tokens/auditor/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

13. Receipt-delivery health

pipelock conductor fleet convergence checks every expected receipt producer against the enrolled-follower roster and its latest accepted signed audit batch. It emits the existing convergence report with delivery_health, plus the last batch ID, signer, schema version, sequence range, dropped count, runtime build, and delivery lag for every expected producer with a batch.

Keep the expected-producer inventory in Git with the deployment declaration. Start from examples/conductor/receipt-producer-inventory.json. Every producer entry needs instance_id and desired_replicas. A local-only producer uses excluded: true, must name both excluded_owner and excluded_reason, and must have a positive desired_replicas value. A producer with desired_replicas: 0 is intentionally scaled down. Both states stay in the report and do not count as delivery failures.

pipelock conductor fleet convergence \
  --inventory /etc/pipelock/conductor/receipt-producers.json \
  --server https://conductor.api.vendor.example:8895 \
  --token-file /etc/pipelock/conductor/tokens/auditor/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

The command exits nonzero after writing the JSON report when an expected producer has no recent accepted batch. Use that exit status for a receipt delivery alert. Enrollment, runtime health, and bundle drift remain separate fields in the same report; they do not change the delivery exit status while a recent batch is still arriving. --require-current=false is for collecting a diagnostic snapshot while an operator works the alert.

Inventory validation happens before the delivery-health check. The command rejects invalid entries, including duplicate producer IDs, missing replica counts, and unexplained local-only exceptions, rather than treating them as healthy. Delivery health then treats a missing, stale, implausibly future-dated, or incomplete accepted-batch record as alerting. A recent timestamp is required, but it does not turn an invalid inventory into healthy delivery evidence.

14. Remove a follower

pipelock conductor follower remove decommissions one exact follower identity. It replaces the active enrollment with an inactive tombstone. The follower disappears from fleet status, and the removed identity can't submit evidence with its enrolled or matching static audit key. The command is admin-only and requires the full org/fleet/instance/environment tuple; a wrong or already-removed follower fails loud instead of being treated as success.

pipelock conductor follower remove \
  --server https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org-id org-acme \
  --fleet-id prod \
  --instance-id edge-01 \
  --environment prod \
  --token-file /etc/pipelock/conductor/tokens/admin/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

Confirm removal:

pipelock conductor fleet status \
  --server https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org-id org-acme --fleet-id prod --instance-id edge-01 \
  --token-file /etc/pipelock/conductor/tokens/admin/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

Expected result: no matching enrolled follower. If the old follower keeps running, it can still enforce local policy, but its later audit batches no longer verify through the Conductor enrollment resolver.

15. Back up and restore Conductor state

Back up the Conductor storage directory as one unit while Conductor is stopped or from a crash-consistent read-only volume snapshot:

pipelock conductor store backup \
  --storage-dir /var/lib/pipelock/conductor \
  --backup-dir /secure-backups/pipelock/conductor-20260622T120000Z

Restore is dry-run by default:

pipelock conductor store restore \
  --storage-dir /var/lib/pipelock/conductor \
  --backup-dir /secure-backups/pipelock/conductor-20260622T120000Z

Apply the restore only while Conductor is stopped:

pipelock conductor store restore \
  --storage-dir /var/lib/pipelock/conductor \
  --backup-dir /secure-backups/pipelock/conductor-20260622T120000Z \
  --confirm

The storage backup includes policy bundles, stream heads, follower enrollments, emergency controls, and audit DB files under --storage-dir. It does not include TLS private keys, operator signing keys, license tokens, or Kubernetes Secrets; restore those through the deployment's secret/KMS process before starting Conductor. See Conductor backup and restore.

After restore:

pipelock conductor store inspect-offline \
  --storage-dir /var/lib/pipelock/conductor

Expected result: the restored policy-bundle store reports the expected stream heads and no startup-bricking orphan records.

16. Query the audit sink

pipelock conductor audit query queries the audit sink's stored evidence metadata for one org/fleet/instance, read over mTLS with an auditor token:

pipelock conductor audit query \
  --server https://conductor.pipelock-control.svc.cluster.local:8895 \
  --org-id org-acme --fleet-id prod --instance-id edge-01 \
  --token-file /etc/pipelock/conductor/tokens/auditor/token \
  --client-cert /etc/pipelock/operator.crt \
  --client-key /etc/pipelock/operator.key \
  --ca-file /etc/pipelock/conductor-ca.pem

The audit query API returns metadata only — it never exports raw stored payload bytes. The sink's --storage-dir is the raw-evidence escrow boundary; treat it as sensitive operator-controlled storage.

Verify follower evidence offline

Every follower emits Ed25519-signed, hash-chained evidence and ships it to the sink in signed batches. An operator with the raw evidence — the follower's own recorder output, or the sink's escrow — verifies it offline against a pinned key, with no trust in Conductor, the follower, or the storage layer:

# Verify a follower's receipt chain against the pinned recorder public key.
# --chain takes the evidence DIRECTORY; --key is a hex key or a key-file path
# (repeat --key across a signing-key rotation, one per trusted segment key).
pipelock verify-receipt --chain /var/lib/pipelock/recorder/evidence \
  --key /etc/pipelock/keys/edge-01-recorder.pub

Without --key, verification is structural-only and exits non-zero unless you pass --allow-unpinned — pin the trusted signer key to get a meaningful CHAIN VALID. A cleanly stopped single-chain run that recorded EvidenceReceipt v2 receipts reports VALID PREFIX, END UNPROVEN and exits 3 in this mode; add --whole-recorder to prove each run's end from its transcript root. A closed multi-chain receipt group is checked as a group and exits 0 with GROUP_VALID. See receipt verification.

17. Rotate certs and keys

The operator-lifecycle rule for every piece of fleet state: prove you can rotate, revoke, and recover it, not just create it.

• TLS certificates (server + follower). cert-manager rotates automatically via renewBefore; the workload picks up the new Secret without a restart. To force a rotation, delete the cert Secret and let cert-manager reissue. The CA stays stable, so no roster change is needed.
• The fleet CA. Rotating the CA is a bigger event: issue the new CA, add it to followers' server_ca_file trust bundle (and the Conductor --client-ca bundle) before cutting over so both old and new chains validate during the overlap, then reissue leaf certs from the new CA and retire the old one.
• Policy-bundle signing key. Generate a new policy-bundle-signing key, rebuild the roster (signing roster build) including the new key (mark the old one status=revoked to retire it), re-sign and redistribute the roster, then publish future bundles with the new key. Followers pin the roster by root fingerprint, so the roster root is the long-lived anchor — keep it offline.
• Threshold keys (kill / rollback). Rotate per approver: generate a fresh approver key, update the roster and the Conductor --trusted-control-key flags, revoke the old one. The threshold means losing one approver key does not compromise the authority.
• Recorder keys. A signing-key rotation does not brick the receipt chain — the emitter opens a new, linked chain segment anchored to the prior tail with a key-transition marker, and verify-receipt --chain is segment-aware. Generate the new recorder key, update the follower config, and the chain stays offline-verifiable across the boundary.
• Operator tokens. Tokens are file-mounted. Rotate by writing a new token to the Secret and restarting the consuming server; there is no in-flight token refresh.

Stale-policy fail-closed enforcement

Under stale_policy.strict_deny_all (the default), a follower re-evaluates its active bundle each poll interval and, once that bundle is missing, unreadable, or past its grace window, engages the independent conductor_stale kill-switch source — denying all traffic across every transport (forward, CONNECT, WebSocket, MCP) until a fresh in-grace bundle applies. The conductor_stale source is OR-composed and independent of the remote-kill source: clearing remote-kill does not clear stale denial, and clearing stale does not clear remote-kill.

On proven license revocation or expiry the follower tears down its Conductor pollers and keeps only free local detection running (re-activatable by restart). Under a strict stale policy, teardown engages the conductor_stale source before the enforcer's ticker stops, so the deny is in place even though no later tick will re-evaluate the bundle — this closes the teardown fail-open window. Under continue_last_known_good the flag is not set and the follower keeps serving its last applied bundle.

Known limitations (be honest about these)

• Several follower conductor: fields are reserved. created_skew_seconds, max_min_version_*_skew, max_capability_threshold, and emergency_stream validate but are not enforced by the follower runtime yet.
• trust-root-rotation and enrollment-token-signing keys are wire-stable purpose bindings with no consuming workflow yet.

See also

• Conductor guide — architecture, planes, licensing, flags
• Conductor dev runbook — local round-trip walkthrough
• Kubernetes enterprise deployment — Helm topology
• Conductor & audit sink design — protocol + storage
• Helm chart README — full values reference
• Configuration reference: Conductor follower
• Receipt verification — verify follower evidence offline
• pipelock license — install and check the fleet license
• Conductor backup and restore — DR commands and checks