Add the sush proxy to the switch zone by plotnick · Pull Request #11253 · oxidecomputer/omicron

plotnick · 2026-09-06T23:13:34Z

Continues the integration of the Support Shell (RFD 620), building on the sled-agent embedding: a new switch zone service, sush-proxy, terminates technician-port connections and routes each request to a sled's sush server. The proxy finds sush servers by probing the bootstrap and underlay prefixes given by DDM, and polls MGS for the cubby map so a request may name its target sled by cubby.

For security review:

On real hardware the proxy serves TLS backed by the sled's platform identity. At zone startup, sled-agent generates an ephemeral key and has the RoT sign its certificate with the TQ key (Ed25519 over the SHA3-256 digest of the TBS certificate), and the sush client verifies that the chain ends at a platform identity root. Simulated and emulated environments have no RoT, so their proxies serve plaintext. This is described in the transport security section of RFD 620.

This continues the integration of the [Support Shell](https://github.com/oxidecomputer/sush) ([RFD 620](https://rfd.shared.oxide.computer/rfd/0620)), building on the sled-agent embedding: a new switch zone service, `sush-proxy`, terminates technician-port connections and routes each request to a sled's sush server. The proxy finds sush servers by probing the bootstrap and underlay prefixes given by DDM, and polls MGS for the cubby map so a request may name its target sled by cubby. **For security review:** On real hardware the proxy serves TLS backed by the sled's platform identity. At zone startup, sled-agent generates an ephemeral key and has the RoT sign its certificate with the TQ key (Ed25519 over the SHA3-256 digest of the TBS certificate), and the sush client verifies that the chain ends at a platform identity root. Simulated and emulated environments have no RoT, so their proxies serve plaintext. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick · 2026-09-09T06:00:12Z

A heads-up: @rmustacc identified a very serious problem in the security story here, which is that we are forbidden by policy (which we're currently ignoring) from signing certificates with the platform id leaf certificate. I will be sketching a work-around tomorrow (Wed, Sep 9), and it will have some knock-on effects (mostly positive, I think).

plotnick · 2026-09-10T04:46:22Z

Heads-up follow-up: I did not get to implement the previously mentioned work-around today (Wed), but I did document it in the transport security section of RFD 620. Feedback would be welcome, and I'll try again to prototype it tomorrow (Thu) based on the sketch there.

…h-proxy Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # workspace-hack/Cargo.toml

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # workspace-hack/Cargo.toml

plotnick · 2026-09-11T15:12:22Z

Follow-up follow-up: the "voucher" workaround is now implemented in oxidecomputer/sush#86. New commits will be pushed here for the Omicron side.

The proxy's ephemeral cert is now self-signed which carries the RoT's voucher for its key (a Trust Quorum signature over the tagged SPKI digest; see RFD 620 §4.6.2.1), rather than claiming tq-dhe as its issuer (which violates critical constraints). The TLS mode is renamed `Platform` → `Vouched` to match. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

internet-diglett

This looks good to me from the networking and switch zone perspective!

jmpesp

I haven't had a chance to review sush_server::ProxyServer yet, but here's a first round of feedback

jmpesp · 2026-09-11T22:37:17Z

+                        .add_property(
+                            "address",
+                            "astring",
+                            &format!("[::]:{SUSH_PROXY_PORT}"),


Do we want to restrict this at all?

Assuming "this" is [::], no, because we need to listen on both the tech port and underlay addresses (for the Nexus tunnel) as they become available. Restriction here would mean plumbing addresses up dynamically for no real win that I can see.

jmpesp · 2026-09-11T22:37:58Z

+const SUSH_PROXY_CERT_VALIDITY: Duration =
+    Duration::from_secs(365 * 24 * 60 * 60);


This is regenerated every sled-agent restart?

Every switch zone startup, which includes sled-agent restarts, yes.

jmpesp · 2026-09-11T22:42:46Z

+                }
+            }
+        }
+        targets.send_modify(|t| t.cubbies.extend(cubbies));


The same comment from the previous PR applies here, w.r.t clobbering the map.

This map has different semantics from the one in '52. There, each round is one complete response from MGS; here, each round polls every SP separately, and we don't want to drop a cubby's entry just because it missed a round. Eventually we'll want to have some kind of stale-entry expiration, but a sled not in this map can't accept jobs targeted by cubby, so we'd like to keep the entries even for a flaky sled (or while it's rebooting, or whatever).

we don't want to drop a cubby's entry just because it missed a round

I think we do, for the same reason as the other PR: moving sleds around in a not-fully-populated rack would mean the sled would appear in multiple different cubbies.

Quite right, thank you. c2ddf78 takes a middle path: missed rounds still keep their entries (I do think the reboot/flaky case is important here), but moved sleds are evicted from their old cubbies. A removed sled still appears in the map, but is harmless; if it isn't there, jobs will fail at forwarding time, and we can still work out an expiration scheme later.

jmpesp · 2026-09-11T22:42:57Z

+                warn!(log, "unable to fetch underlay prefixes"; "error" => %err)
+            }
+        }
+        targets.send_modify(|t| t.sleds.extend(sleds));


Same situation as above, I think merging into the map is correct here.

hawkw · 2026-09-11T22:59:40Z

+/// Path inside the switch zone to the sush proxy's TLS private key.
+pub const SUSH_PROXY_KEY_PATH: &str = "/etc/sush-proxy/key.pem";
+
+/// Path inside the switch zone to the sush proxy's TLS certificate chain.
+pub const SUSH_PROXY_CERT_CHAIN_PATH: &str = "/etc/sush-proxy/chain.pem";


~~what, if anything, can or should reasonably done to restrict the accessibility of these paths to to processes other than the sush proxy?~~

EDIT: oh, cool, I see that we have a write_private function which ensures the key has the correct permissions. maybe worth noting that in the comment?

Sure, noted in b237150.

hawkw · 2026-09-11T23:08:23Z

+        .create(true)
+        .write(true)
+        .truncate(true)
+        .mode(0o600)


is this sufficient to ensure that only sush-proxy can read the file? what user do other switch zone processes run as? and all SSH sessions presently log in as root, right?

As far as I can tell, everything in the switch zone currently runs as root, yes. You are correct that this proxy could run as a dedicated non-privileged user, and probably drop other capabilities and permissions. I would be happy to take that as a follow-up if you think it's worth it (it may be, I'm not sure). But stealing this key would only let an attacker impersonate a proxy; it would not grant any access to run or examine jobs.

Yeah, for what it's worth, I totally agree that not having everything run as root is very much out of scope for this PR. But, it feels like something we should be thinking about separately. My point was more that the fact that everything, including SSH sessions, runs as root means that restricting who can access the file is currently mostly ceremonial, since basically every process will be running as root anyway. I agree that it's better than not doing it though!

hawkw · 2026-09-11T23:08:51Z

+    /// A vouched proxy identity: an ephemeral key that sled-agent
+    /// generates and the RoT vouches for, served from local files
+    /// along with the platform identity chain that validates it.


would like this to be declauded

Thanks, rewritten in 71a9dc8.

hawkw · 2026-09-11T23:11:16Z

+    tokio::join!(sleds(log, ddm, &tx_targets), cubbies(log, mgs, &tx_targets),);
+    unreachable!("discovery loops never return");


if they never return, why bother joining them? perhaps they should be spawned instead?

(what i am actually saying here is in part that i am a bit concerned about these running in the same task rather than in true parallelism, especially since they both contend the watch channel sender which is morally equivalent to a mutex)

in fact, upon thinking through this further, i don't really get why Targets must be a single watch channel, given that there are two distinct senders which both only touch half of it. It seems like Targets should really be internally a watch channel of cubbies and separately a watch channel of sleds, and then consumers would select either of them having changed

Ok, in order: we join so that panics propagate, kill the proxy, and SMF will restart it. The alternative is to continue serving with stale routes, which could wedge it closed.

On contention: yes, it's morally a mutex, but the write locks are once per 30-second poll round, and the reads are one borrow per request, which consults both maps. I don't think splitting it would really improve much, though we certainly can if you feel strongly about it.

On concurrency: there's no real work happening in either loop, they're just waiting for timeouts and I/O.

Ok, in order: we join so that panics propagate, kill the proxy, and SMF will restart it. The alternative is to continue serving with stale routes, which could wedge it closed.

Are we building with panic = "abort"? If so, this shouldn't be necessary.

On contention: yes, it's morally a mutex, but the write locks are once per 30-second poll round, and the reads are one borrow per request, which consults both maps. I don't think splitting it would really improve much, though we certainly can if you feel strongly about it.

On concurrency: there's no real work happening in either loop, they're just waiting for timeouts and I/O.

My concern here is just that having two separate concurrent futures in the same task which may both try to write lock the same mutex feels sketchy. I agree that there is no risk of deadlock in the current code, but if some future thing comes along and makes one of the poll loops also try to read from the watch channel, it could become a problem. I don't feel that strongly about it, it's just mildly concerning for the future.

Ok, you've convinced me, thanks for hanging in there, and for the reminder about panic behavior. I believe that d615d83 should do what you're looking for: it spawns two independent tasks, one for each discovery loop, and bails if either ever returns. Depends on sush e96e719.

Thanks, this feels neater. Again, I wasn't particularly concerned about a bug in the previous code, per se, but that it might be a bit too easy to introduce one if it grew more complex later. The new version makes a lot of those problems impossible.

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Move the proxy onto sush's own sush-proxy crate, renaming this package to omicron-sush-proxy to avoid the clash. Co-authored-by: Claude <noreply@anthropic.com>

Polling all 32 cubbies is slow (minutes), especially when most of them are empty (as on a racklette), because MGS takes seconds to give up on each empty cubby. So instead, use the strategy that other inventory collectors use: ask MGS for the ignition list, and only ask about cubbies where ignition reports a running sled. Merge each round into the cubby map, dropping cubbies that ignition reports empty. sled-agent now looks for MGS at every candidate address concurrently, since requests to an address with no MGS hang until they time out. Also share the polling code between the proxy and sled-agent, which is why the API dependencies changed. And clean up and simplify helpers, and don't panic on HTTP client errors in the proxy. Co-authored-by: Claude <noreply@anthropic.com>

Bring in the production and lab job signing keys, replacing the sandbox keys. Co-authored-by: Claude <noreply@anthropic.com>

Co-authored-by: Claude <noreply@anthropic.com>

Pick up main's move to SP release 1.86.0, which fixes the Cosmo host flash fault. Co-authored-by: Claude <noreply@anthropic.com>

Move sush-proxy to the same sush main commit as the other sush crates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>

jmpesp · 2026-11-06T21:51:33Z

+                        )
+                    })
+                    .collect();
+                let mut polls: FuturesUnordered<_> = mgs_clients


I'd use ParallelTaskSet here, FuturesUnordered does not execute in parallel.

Thanks for the suggestion, I hadn't seen ParallelTaskSet before. But I think FuturesUnordered is actually the right choice here. First, these are just network requests, so concurrency is what matters, not parallelism. Second, ParallelTaskSet seems designed to cap the number of tasks running at once; but launching these all at once is essential, because a request to a sled without a switch zone hangs until it times out, and requests to the actual scrimlets could get stuck behind those. Finally, it would need 'static futures, which I prefer to avoid if possible.

jmpesp · 2026-11-06T21:54:05Z

+                            "no SP state for cubby";
+                            "cubby" => cubby,
+                            "error" => %err,
+                        );


What about a third state, like SledState::Off ?

Otherwise this sled would simply be missing

If the sled is powered off via ignition, its SP can't give us its baseboard ID, so an Off state couldn't add any entries to the cubby map. All it could do is distinguish a sled that ignition has powered off from one whose SP isn't answering (which is usually a transient condition, like an SP reset). Either way, the sled isn't reachable, so the client would report the same error.

Co-authored-by: Claude <noreply@anthropic.com> # Conflicts: # Cargo.toml

plotnick mentioned this pull request Sep 6, 2026

Tunnel sush through Nexus #11254

Open

plotnick force-pushed the sush-sled-agent branch from e7b2c72 to 01f98ba Compare September 6, 2026 23:53

plotnick force-pushed the sush-switch-proxy branch from 0ce2c9d to 7dc9f6d Compare September 6, 2026 23:54

internet-diglett self-requested a review September 8, 2026 17:42

Merge branch 'sush-sled-agent' into sush-switch-proxy

0603074

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick added 2 commits September 10, 2026 17:30

Merge remote-tracking branch 'github/sush-sled-agent' into sush-switc…

588a9b7

…h-proxy Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # workspace-hack/Cargo.toml

Merge branch 'sush-sled-agent' into sush-switch-proxy

75ec5e2

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # workspace-hack/Cargo.toml

internet-diglett approved these changes Sep 11, 2026

View reviewed changes

jmpesp requested changes Sep 11, 2026

View reviewed changes

hawkw reviewed Sep 11, 2026

View reviewed changes

Comment thread sush-proxy/src/lib.rs Resolved

Comment thread sush-proxy/src/lib.rs Resolved

Comment thread sush-proxy/src/lib.rs Resolved

Comment thread sush-proxy/src/main.rs Resolved

plotnick and others added 5 commits September 12, 2026 08:01

Move sush-proxy ipcc comment

80fb296

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

De-splice comment

f8416df

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Inline single-use helper fn

1f97d71

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Less syntax per line

41a6e5b

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

wrap me please

ce874f4

Co-authored-by: Eliza Weisman <eliza@elizas.website> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick force-pushed the sush-switch-proxy branch from 26edb6a to ce874f4 Compare September 12, 2026 14:10

plotnick and others added 4 commits September 12, 2026 08:43

Rename Vouched → RotVouched

c249168

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Note file privacy

b237150

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Ephemeral proxy certs never expire

d319f18

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Declaude Tls variant docs

71a9dc8

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick force-pushed the sush-switch-proxy branch from 33fa5ce to 71a9dc8 Compare September 12, 2026 15:15

plotnick and others added 2 commits September 12, 2026 09:34

Clarify Args::home doc

5e5a39e

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Spawn sled & cubby discovery loops as independent tasks

d615d83

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick force-pushed the sush-switch-proxy branch from d650a9a to d615d83 Compare September 12, 2026 20:53

Handle moved sleds in MGS probe

c2ddf78

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

plotnick mentioned this pull request Oct 23, 2026

Access to switch-zone proxy via tech port w/o unlock oxidecomputer/sush#27

Closed

plotnick and others added 7 commits November 5, 2026 08:58

Merge branch 'sush-sled-agent' into sush-switch-proxy

8ba1b62

Move the proxy onto sush's own sush-proxy crate, renaming this package to omicron-sush-proxy to avoid the clash. Co-authored-by: Claude <noreply@anthropic.com>

Merge branch 'sush-sled-agent' into sush-switch-proxy

ad4101a

Bring in the production and lab job signing keys, replacing the sandbox keys. Co-authored-by: Claude <noreply@anthropic.com>

Merge branch 'sush-sled-agent' into sush-switch-proxy

88ff83a

Co-authored-by: Claude <noreply@anthropic.com>

Probe only the bootstrap network for servers

7fb466e

Co-authored-by: Claude <noreply@anthropic.com>

Merge branch 'sush-sled-agent' into sush-switch-proxy

80e8b32

Pick up main's move to SP release 1.86.0, which fixes the Cosmo host flash fault. Co-authored-by: Claude <noreply@anthropic.com>

Merge branch 'sush-sled-agent' into sush-switch-proxy

f7ebdfa

Move sush-proxy to the same sush main commit as the other sush crates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>

jmpesp reviewed Nov 6, 2026

View reviewed changes

plotnick marked this pull request as ready for review November 6, 2026 22:47

Merge branch 'sush-sled-agent' into sush-switch-proxy

39f2a2f

Co-authored-by: Claude <noreply@anthropic.com> # Conflicts: # Cargo.toml

		const SUSH_PROXY_CERT_VALIDITY: Duration =
		Duration::from_secs(365 * 24 * 60 * 60);

		tokio::join!(sleds(log, ddm, &tx_targets), cubbies(log, mgs, &tx_targets),);
		unreachable!("discovery loops never return");

Conversation

plotnick commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

plotnick commented Sep 9, 2026

Uh oh!

plotnick commented Sep 10, 2026

Uh oh!

plotnick commented Sep 11, 2026

Uh oh!

internet-diglett left a comment

Choose a reason for hiding this comment

Uh oh!

jmpesp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plotnick Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plotnick commented Sep 6, 2026 •

edited

Loading

plotnick Sep 12, 2026 •

edited

Loading