Remove upsert_sled in favor of server versioned API and polling #11627

Activity

1. andrewjstone

   self-assigned this

   on Oct 28, 2026
2. andrewjstone

   added

   Sled AgentRelated to the Per-Sled Configuration and ManagementRelated to the Per-Sled Configuration and Management

   nexusRelated to nexusRelated to nexus

   multi-rack

   on Oct 28, 2026
3. bnaecker commented on Oct 29, 2026

   

   bnaecker

   on Oct 29, 2026

   Collaborator

   More actions

   Related #7980
4. davepacheco commented on Oct 30, 2026

   

   davepacheco

   on Oct 30, 2026

   Collaborator

   More actions

   I gather from today's update watercooler that there's a plan in-progress to unblock multi-rack. That's all good but I wanted to jot down some notes here about longer-term fixes.

   ---

   I believe this API endpoint is from very early days, before sled lifecycle was much of a thing, before RSS, and before "sled add". This API serves a couple of different purposes:

   1. It's the first place Nexus learns about a sled and some of its static information, including its IP. The fact that Nexus doesn't know about sleds prior to their first attempted upsert is part of what makes fixing this tricky (since Nexus can't reach out to the sled until it knows this)
   2. It's the place where some dynamic state about the system is reported (e.g., reservoir size)
   3. It's the point where Nexus learns that Sled Agent has restarted (possibly as a result of a system reboot). This is an important signal for Nexus to go check that the sled is doing what it's supposed to be doing (e.g., running the instances that are expected).

   It may be helpful to consider these use cases separately.

   The static information (e.g., sled type, memory, IP address) ought to be known even before Sled Agent starts. In the single-rack case, RSS could know this (if it doesn't already) and could provide it in the handoff request. In the case of a sled being added to the control plane, similarly, this information could be inserted as part of the "sled add" process. I don't know enough about how multi-rack join works to suggest what the analogous thing would be but I would guess there's a similar point where this information can be told to Nexus.

   The dynamic information (e.g., reservoir size) seems like a good case for inventory, which already runs periodically and on-demand when we think it's changed.

   So in the longer term, maybe something like this:

   1. RSS, "sled add", and "multirack join" would be responsible for collecting the static information and providing it to Nexus for it to insert the initial sled record.
   2. Dynamic stuff like inventory and whether it's a scrimlet could move to inventory. (Note that we could still have something copy it from inventory to the sled table, if it makes sense to latch the last-seen state, or if it's too painful to migrate consumers of this data.)
   3. I assume we'd still keep a client-side-versioned upcall, but all it would do is activate some background tasks (like inventory collection, to catch any updated dynamic state, or instance_watcher, to make sure the instances are running).
5. andrewjstone commented on Oct 30, 2026

   

   andrewjstone

   on Oct 30, 2026

   ContributorAuthor

   More actions

   I gather from today's update watercooler that there's a plan in-progress to unblock multi-rack. That's all good but I wanted to jot down some notes here about longer-term fixes.

   Indeed. I wrote this up before that discussion. The current plan is to continue to use the client-side upcall as a "doorbell" and then call down to sled-agent to fetch all this information. The code will actually use the current structure as a doorbell and ignore it once nexus is upgraded. That will be good enough for multirack since multirack won't be functional until after the upgrade. Then the upcall will be changed to be a pure doorbell and remove the existing info passed upwards. @sunshowers deserves all the credit for thinking of this solution.

   I believe this API endpoint is from very early days, before sled lifecycle was much of a thing, before RSS, and before "sled add". This API serves a couple of different purposes:

   1. It's the first place Nexus learns about a sled and some of its static information, including its IP.  The fact that Nexus doesn't know about sleds prior to their first attempted upsert is part of what makes fixing this tricky (since Nexus can't reach out to the sled until it knows this)
   

   Yep. I was calling this the "discovery" phase. When I wrote this issue my plan was to utilize a union of the sled_underlay_subnet_allocation table and existing sled table to find the the sled-agents. If we keep the doorbell, this is no longer necessary, which is quite nice.

   2. It's the place where some dynamic state about the system is reported (e.g., reservoir size)
   

   Indeed. We can still use the doorbell and poll for this.

   3. It's the point where Nexus learns that Sled Agent has restarted (possibly as a result of a system reboot).  This is an important signal for Nexus to go check that the sled is doing what it's supposed to be doing (e.g., running the instances that are expected).
   

   Same as above.

   It may be helpful to consider these use cases separately.

   The static information (e.g., sled type, memory, IP address) ought to be known even before Sled Agent starts. In the single-rack case, RSS could know this (if it doesn't already) and could provide it in the handoff request. In the case of a sled being added to the control plane, similarly, this information could be inserted as part of the "sled add" process. I don't know enough about how multi-rack join works to suggest what the analogous thing would be but I would guess there's a similar point where this information can be told to Nexus.

   The dynamic information (e.g., reservoir size) seems like a good case for inventory, which already runs periodically and on-demand when we think it's changed.

   So in the longer term, maybe something like this:

   1. RSS, "sled add", and "multirack join" would be responsible for collecting the static information and providing it to Nexus for it to insert the initial `sled` record.
   
   2. Dynamic stuff like inventory and whether it's a scrimlet could move to inventory.  (Note that we could still have something copy it from inventory to the `sled` table, if it makes sense to latch the last-seen state, or if it's too painful to migrate consumers of this data.)
   
   3. I assume we'd still keep a client-side-versioned upcall, but all it would do is activate some background tasks (like inventory collection, to catch any updated dynamic state, or instance_watcher, to make sure the instances are running).
   

   I think it would be useful to separate out the sled table from the inventory related information, but I'm no longer that concerned about it. The doorbell + polling seems like a good solution that can last us a long time.

   👍React with 👍1Reacted by David Pacheco
6. andrewjstone

   assigned

   sunshowers

   and unassigned

   andrewjstone

   on Oct 30, 2026