chore(poc): dev SIGUSR2 upgrade trigger + evidence notes

Not for merge — scaffolding kept separate at the top of the branch: a
SIGUSR2/marker-file manual upgrade trigger in envd main (dev convenience,
inert in prod without the marker; superseded by the orchestrator trigger),
and POC-NOTES.md, the running evidence log.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: kalyazin

POC-NOTES.md

@@ -0,0 +1,141 @@
+# envd live-upgrade PoC — running notes / evidence
+
+Branch: `poc/envd-live-upgrade`. Design: `~/dev/reports/2026-06-22-envd-live-upgrade-design.md` §11.
+Milestone 1 = feasibility + risk register; STOP for go/no-go before Milestone 2.
+
+## Risk register (live)
+
+| ID | Risk | Status | Evidence |
+| --- | --- | --- | --- |
+| R-A | stdio + PTY fds survive same-PID execve, no SIGPIPE | **confirmed-benign** | E0 |
+| R-B | pidfd reaping works, no os/exec collision | **confirmed-benign** (spike); in-guest pending | E0 (E2 to confirm no-collision) |
+| R-C | execve invisible to systemd Type=simple | partially (PID continuity proven); systemd pending | E0 (E1 to confirm under systemd) |
+| (impl) | in-guest handover patch (process path) | **written, compiles, vet-clean, committed** | commit 5706657a1 |
+| R-D | freeze covers workload; CLOEXEC race closed | **confirmed e2e** (workload frozen+survived swap); explicit socat-leak (E3) still pending | E7 |
+| R-E | process table reconstructs (List/Connect) | **confirmed e2e** (re-adopted pid in List post-swap) | E7 |
+
+## E7 — e2e on dev cluster (real guest 6.1, real SDK) — ALL PASS (2026-06-23)
+
+Patched envd v0.6.7 deployed to dev (`gs://e2b-dev-nikita-fc-env-pipeline/envd`), baked into
+template `envd-upgrade-poc` (`envdVersion: 0.6.7` confirmed via API). Driver:
+`scratchpad/e2e_upgrade.py`. Create sandbox → start a stdout-writing workload (so a broken pipe
+would SIGPIPE-kill it) → `touch /run/e2b/allow-upgrade` + `kill -USR2 $(pidof envd)` → reconnect.
+
+```
+sandbox ilqawa4yqv9l1543gr3me
+workload pid 1264
+pre-upgrade:  logLines=10 envdPid=1224 bakedEnvd=0.6.7
+envd journal: "envd: SIGUSR2 -> self-upgrade (from v0.6.7)"   # execve actually fired
+post-upgrade: alive=ALIVE logLines=33 envdPid=1224 listPids=[1264]
+  [PASS] swap actually fired (journal)
+  [PASS] R-A/R-D: stdout-writing workload survived the swap
+  [PASS] workload kept running across swap (log 10->33)
+  [PASS] R-C: envd same PID across execve (1224==1224)
+  [PASS] R-E: workload re-adopted into new envd's List
+  RESULT: ALL PASS
+```
+
+Decisive: the handover mechanism works end-to-end on a real cluster guest. The execve fired
+(journal), the running stdout-writing workload survived with no SIGPIPE and kept producing
+output across the swap, envd kept its PID, the workload was re-adopted into the new image's
+process table, and *new* commands (the post-swap `wc`/`kill -0`) ran fine — exercising os/exec
+reaping alongside the pidfd-reaped re-adopted process (R-B no-collision).
+
+Not yet run: E2 gap-exit exit-code case, E3 explicit socat-fd-leak, E4 watchers (R-F deferred),
+E5 plain-resume baseline (Q1), Q3 telemetry. None blocking the verdict.
+
+**Workload & coverage.** Every cluster run used a synthetic counter loop, background/root:
+`i=0; while true; do echo "out$i"; echo "$i" >> /tmp/poc.log; i=$((i+1)); sleep 0.3; done`. The
+stdout `echo` exercises the envd-owned pipe (failed fd-carry → SIGPIPE death); the tmpfs file is
+an out-of-band liveness probe (line growth = survived + still running). Proxy for an arbitrary
+process, not a real app. Cluster runs = non-PTY stdout; PTY validated only in E0 spike; stdin /
+native long-running / multi-subscriber streams untested.
+
+## M2 — production trigger path (2026-06-24)
+
+Goal: real cross-version upgrade (vA->vB) with host-side binary delivery, wired to the resume
+path. Built envd `/upgrade` endpoint (POST, target path in `X-Envd-Upgrade-Bin`; SIGUSR2 also
+reads `/run/e2b/upgrade-to`) + orchestrator `CallEnvdUpgrade` trigger helper (committed).
+
+**vA->vB cross-version e2e — ALL PASS** (`scratchpad/e2e_vab.py`, sandbox ifea94q6di0wy79lctvxh):
+vA=0.6.7 baked in template; deployed/delivered vB=0.6.8. Delivered vB into the guest (host-side),
+pointed envd at it, triggered upgrade.
+```
+pre:  envdPid=1220 runningEnvd=vA(0.6.7)  logLines=11
+journal: "self-upgrade (from v0.6.7, newBin=/usr/bin/envd.next)"
+journal: "resumed as v0.6.8 after handover (from v0.6.7, 1 procs)"
+post: alive=ALIVE logLines=47 envdPid=1220 listPids=[1264]
+  RESULT: ALL PASS — running envd flipped 0.6.7->0.6.8, workload survived, same PID, re-adopted
+```
+The running envd image genuinely changed version while the live workload was preserved — the
+production mechanism (host-delivered different binary + targeted upgrade). Same PID across execve
+confirmed via `pidof envd.next`==prior `pidof envd` (comm changed to the new binary's basename).
+
+**Orchestrator auto-trigger (resume path):** trigger helper `Sandbox.CallEnvdUpgrade` committed
+(direct POST /upgrade + WaitForEnvd). Remaining for a full auto-resume deploy:
+- insertion point: `server/sandboxes.go` resume handler (after `ResumeSandbox`, ~line 763) where
+  `s.proxy` (SandboxProxy, for `sandboxtools.CopyFile` delivery) and `s.config.HostEnvdPath` are
+  in scope; resolver compares built-with `config.Envd.Version` vs target.
+- **caveat:** the target envd binary isn't guaranteed present on client nodes (HOST_ENVD_PATH is
+  a build-node path) — for the PoC it's pre-staged at `/opt/poc-envd-target` on the n2 nodes.
+
+### M2 AUTO-RESUME e2e — ALL PASS (2026-06-24, deployed to dev)
+
+Deployed patched orchestrator to dev (binary hot-swap on n2 nodes; backup at
+`gs://…/orchestrator.poc-backup`) + patched envd 0.6.7 (body-handling `/upgrade`) baked into the
+template + vB 0.6.8 staged at `/opt/poc-envd-target` on n2. `scratchpad/e2e_auto.py` (sandbox
+izvhctu5n0ja8vj8rb719): create 0.6.7 sandbox + workload → **pause → resume** with NO manual
+trigger.
+```
+journal: self-upgrade (from v0.6.7, newBin="/usr/bin/envd.next")
+journal: resumed as v0.6.8 after handover (from v0.6.7, 1 procs)
+post: alive=ALIVE log 9->23 re-adopted in List
+RESULT: ALL PASS
+```
+The ORCHESTRATOR auto-delivered+triggered the upgrade at resume (Loki: maybeUpgradeEnvd ->
+"delivering+triggering"); running envd flipped 0.6.7->0.6.8, workload preserved.
+
+Two debugging finds: (1) hook must be in the **Create** handler (API /resume = Create
+snapshot=true), not Checkpoint; (2) delivery can't use build-time `CopyFile` (posts to envd
+`/files` unauthenticated → runtime sandbox rejects with "closed pipe") — stream the binary in
+the body of the **token-authenticated** `/upgrade` call instead.
+
+**Cluster side effects to clean up:** patched orchestrator on n2 (revert: copy
+`orchestrator.poc-backup` back + reroll); patched envd 0.6.7 active (revert: `build-and-upload/envd`
+from main); `/opt/poc-envd-target` staged on n2; **n4 drain status was wiped by the orchestrator
+restarts** (n4 now `ready` — normally kept draining for TSC portability; needs re-draining).
+| R-E | process table reconstructs (List/Connect) | pending | E1 |
+| R-F | watcher re-arm Option C | pending | E4 |
+| R-G | access_token_hash carry closes 401 window | pending | E1 |
+
+## E0 — mechanism spike (laptop, host kernel 6.17, Go 1.26.3) — ALL PASS
+
+`packages/envd/cmd/upgrade-spike/main.go`. Two-phase same-PID `syscall.Exec`.
+
+```
+[phase1] pid=3334504
+[phase1] pre-exec read from A: "A0\n"
+[phase1] exec -> phase2 (carry pipeA=100 pidA=101 pidB=102 lis=103 pty=104)
+[phase2] pid=3334504 (same PID => R-C ok)
+[phase2] post-exec read from A:  [A1 A2 A3]      # pipe child survived + streamed
+[phase2] post-exec read from PTY:[P0 P1 P2]      # PTY child survived + streamed
+[phase2] control child B: wait4=... signaled=true sig=broken pipe   # NEGATIVE CONTROL
+[phase2] pidfd A readable; wait4=... signaled=true sig=terminated    # pidfd reaping
+RESULT: ALL PASS
+```
+
+Findings:
+- **R-A pipe + PTY:** carried read-end / master across execve; children kept writing, no SIGPIPE.
+- **Negative control:** child B whose read-end was *not* carried died of SIGPIPE on its next
+  write — proves the fd-carry (CLOEXEC-clear + dup3) is what saves the workload, not luck.
+- **R-C:** PID identical across phases (`syscall.Exec` keeps PID; systemd would not observe it).
+- **listener:** inherited TCP listener fd reconstructed via `net.FileListener` and Accept()ed.
+- **R-B:** `pidfd_open` fd became POLLIN-readable on child exit; targeted `wait4(pid)` harvested
+  the status — the no-`wait4(-1)` path. (os/exec-coexistence still to confirm in-guest, E2.)
+
+Mechanics that mattered: `unix.Dup3(old, target, 0)` to fixed high fd numbers (100-104) clears
+CLOEXEC; done under `syscall.ForkLock`. No Siginfo parsing needed — poll(pidfd)+pid-specific
+wait4 is enough to prove the targeted reap.
+
+Verdict: the two highest-risk assumptions (fd inheritance w/o SIGPIPE; pidfd reaping) hold on a
+real kernel. Proceed to in-guest (E1+) — no cheap no-go here.

packages/envd/main.go

@@ -8,8 +8,11 @@ import (
 	"log"
 	"net/http"
 	"os"
+	"os/signal"
 	"path/filepath"
 	"strconv"
+	"strings"
+	"syscall"
 	"time"
 
 	"connectrpc.com/authn"
@@ -294,6 +297,29 @@ func run() error {
 		return nil
 	}
 
+	// Manual trigger via SIGUSR2 (dev convenience): inert unless the per-sandbox
+	// marker /run/e2b/allow-upgrade exists, so deploying this envd doesn't arm
+	// every dev sandbox. If /run/e2b/upgrade-to holds a path, that binary is
+	// exec'd (real vA->vB); otherwise envd re-execs itself.
+	go func() {
+		ch := make(chan os.Signal, 1)
+		signal.Notify(ch, syscall.SIGUSR2)
+		for range ch {
+			if _, err := os.Stat("/run/e2b/allow-upgrade"); err != nil {
+				fmt.Fprintf(os.Stderr, "envd: SIGUSR2 ignored (no /run/e2b/allow-upgrade marker)\n")
+
+				continue
+			}
+			newBin := ""
+			if b, err := os.ReadFile("/run/e2b/upgrade-to"); err == nil {
+				newBin = strings.TrimSpace(string(b))
+			}
+			if err := doUpgrade(newBin); err != nil {
+				fmt.Fprintf(os.Stderr, "self-upgrade failed: %v\n", err)
+			}
+		}
+	}()
+
 	// Orchestrator-driven trigger: authenticated POST /upgrade with the target
 	// binary path in the X-Envd-Upgrade-Bin header (empty = re-exec self). This
 	// is the production trigger the orchestrator calls at resume after delivering