Repository navigation
"sled add" could use a guard against attempting to add a previously-used sled that has not been clean slated #11671
Description
I'm not sure what would happen if you tried to add a sled that had not been clean slated, but I don't think it would be good. We should make sure this somehow fails gracefully and in a way that can be observed by the user.
Activity
davepacheco commentedon Nov 3, 2026
I'm not clear how bad it would be to insert a non-clean-slated but expunged sled, possibly from another rack, into a rack.
I did just test this on
berlinby:committedThe TQ config has been
Committingfor almost 10 minutes now. (Is this stuck forever? Is there a way out of this?) The sled is reporting this in the sled agent log:The sled brought up its old underlay IP:
(I was doing a bunch of expunge/add testing here.)
That seems bad. (What if this IP were in use by a sled in this rack?) Is its boostrap IP unique? At least it's not able to unlock its local storage and start control plane components.
davepacheco commentedon Nov 3, 2026
I guessed maybe the live-but-not-working sled was holding up the TQ change, but having let it sit overnight and then powered off the sled, it's still stuck
Committing.jgallagher commentedon Nov 3, 2026
No argument that it seems bad, but I don't think the IP can be in use - we never reuse sled subnet reservations (at least today, which means we have a hard cap on how many sleds can added to a rack over its lifetime):
omicron/nexus/db-queries/src/db/datastore/rack.rs
Lines 386 to 392 in 40b22e5
Yes; bootstrap addrs are derived from the NIC's MAC:
omicron/sled-hardware/src/underlay.rs
Lines 110 to 120 in 40b22e5
jgallagher commentedon Nov 3, 2026
@andrewjstone would know for sure, but I think you're going to be stuck in committing until either:
because the sled you're trying to add has to participate in order for the current epoch (which adds it) to commit?
davepacheco commentedon Nov 3, 2026
I think this is still a risk (probably small) if it comes from another rack.
Ah, that makes sense. I did successfully remove it from TQ and that epoch committed.
Since the sled knows it's been expunged from another TQ, I wonder if it could communicate this to Nexus and have Nexus abandon the attempt to add it in that case?
I'm also not sure how any of this works if the sled was not aware that it had been expunged.
Gate sled-agent startup on trust quorum unlock
andrewjstone commentedon Nov 3, 2026
Sorry for not commenting earlier. I've been digging into the larger problem of why the sled-agent is running once a sled has been expunged and rebooted. I think #11684 should fix it.
Sled-agent really shouldn't be running after an expunge and reboot. So it can't talk to Nexus at all. We may be able to get it's status via
omdb nexus trust-quorum proxy-statusbut that's not great from a customer perspective.andrewjstone commentedon Nov 3, 2026
I tested this on voxel. I started with a 4 node cluster, then removed
g2from the trust quorum. I waited for it to commit.Then I logged into
g2, which I had not yet rebooted, and checked the status of sled-agent with the new bootstrap API endpoint and the ipadm state, and rebooted:I then logged back in and did the same thing. The gate held and eventually the state transitioned to
expunged. I added some spaces for clarity.andrewjstone commentedon Nov 3, 2026
I realized that the testing I posted above was for #11684