The MUPdate resolution flow described in RFD 556 assumes that sled agent will reject new sled configs from Nexus if (1) it has detected locally that a MUPdate override is present, and (2) the new sled config does not instruct it to remove a MUPdate override with the same UUID that its locally-detected MUPdate has. (section 2.2 "Implementation", under PUT /omicron-config)
This is necessary to avoid a race where:
- system is MUPdated to release N
- Nexus removes MUPdate override everywhere (replacing all artifact references to "install dataset", etc.)
- customer uploads TUF repo for release N
- Nexus starts sending new sled configs that resolve the install dataset contents to specific artifacts
- system is MUPdated to release N + 1
At this point, when the system comes up, Nexus will still be trying to send sleds configs that resolve install dataset contents to artifacts from release N. That's wrong, though: the install dataset now contains N + 1. This would effectively downgrade all zones in the system (and maybe host OS phase 2?). This would be very bad.
The "remove_mupdate_override" field in the sled config is supposed to prevent this by having sled agent realize that Nexus hasn't noticed the second MUPdate yet and so wait for it to do so.
While working on oxidecomputer/customer-support#1372, I couldn't find code that implements this. I haven't reproduced this problem so I may have just misread the code. But it definitely looked like we'd at least ledger the wrong config if this happened, which seems bad.
The MUPdate resolution flow described in RFD 556 assumes that sled agent will reject new sled configs from Nexus if (1) it has detected locally that a MUPdate override is present, and (2) the new sled config does not instruct it to remove a MUPdate override with the same UUID that its locally-detected MUPdate has. (section 2.2 "Implementation", under
PUT /omicron-config)This is necessary to avoid a race where:
At this point, when the system comes up, Nexus will still be trying to send sleds configs that resolve install dataset contents to artifacts from release N. That's wrong, though: the install dataset now contains N + 1. This would effectively downgrade all zones in the system (and maybe host OS phase 2?). This would be very bad.
The "remove_mupdate_override" field in the sled config is supposed to prevent this by having sled agent realize that Nexus hasn't noticed the second MUPdate yet and so wait for it to do so.
While working on oxidecomputer/customer-support#1372, I couldn't find code that implements this. I haven't reproduced this problem so I may have just misread the code. But it definitely looked like we'd at least ledger the wrong config if this happened, which seems bad.