Repository navigation
want "SMF services in maintenance" under active problems #11675
Description
Once this is possible, we'll want an Active Problem to represent "an SMF service is in maintenance". It should probably name the SMF service (and probably the sled and zone), probably under some "details" section that a customer isn't expected to understand. The only action for the customer is probably "call support". Despite customers not being expected to know these implementation details, there's value in support being able to access this basic information direction in the external API (through the customer?).
This has been a problem specifically lately for https://github.com/oxidecomputer/customer-support/issues/1317, which has been seen a number of times and resulted in the update system's contact_support: true flag being set. The hope is that an active problem can better communicate what's going on here. (The goal would eventually be that all the reasons this flag can light up would trigger active problems and we'd remove the flag.)
Activity
hawkw commentedon Nov 3, 2026
We might want to consider keeping something in the update system APIs but maybe replacing the flag (eventually) with some list of "here are the UUIDs of the active problems that may impact a software update"?
hawkw commentedon Nov 3, 2026
In order to implement this, I think we will want to start by adding a diagnosis engine that actually reads the
SvcsEnabledNotOnlinevalues from the current inventory collection, and opens/closes cases for this. We can do that even in the absence of the APIs for diagnosis engines to create Active Problems, for now those cases would just not be externally visible, but we could test that DE by comparing it to thecontact_supportflag. That part can be decoupled from actually creating the active problem.hawkw commentedon Nov 3, 2026
Re:
Looking at the other checks that are currently performed in
UpdateContactSupportChecks::problems(), I see checks for the followingWe already have a diagnosis engine for stuck sagas (thanks to @smklein in #10592), and one for unhealthy zpools (see #105411). So, we should already be set up to start diagnosing active problems for both of those conditions once the active problems API work lands. Besides SMF services in maintenance, the remaining things we don't already have FM DEs for seem to be the update activity state and sleds with missing update status checks. I think it probably makes sense to group both of those into an "update" diagnosis engine.
Footnotes
Though note that my understanding is that the check for degraded zpools will not actually detect any physical disk failures in practice, since there's some host OS work requried in order to actually mark single-disk zpools as degraded. ↩
smklein commentedon Nov 5, 2026
I can take a swing at getting a PR up for the "case" side of this, though the "problem" side is blocked by @hawkw 's work (as already indicated on this issue).
hawkw commentedon Nov 6, 2026
In 4c0b4ff, I added a very rough initial draft of documentation for this active problem --- this is intended partially as a placeholder and I'm hoping we can expand upon these docs in the future, but I'd love any feedback on it regardless.
davepacheco commentedon Nov 7, 2026
The text here is pretty good, but I believe it's possible that instances and networking could be affected as well. (e.g., if dendrite or ndp is in maintenance in a switch zone)
hawkw commentedon Nov 7, 2026
I'll add that as well. Mainly, what I wanted to get a second opinion on was the level of detail --- I wanted the operator to understand the problem and what its downstream consequences might be, without having to understand the details of SMF or what its different service states mean. Thanks for taking a look!