Nexus should explicitly abandon sagas that it fails to recover for non-transient reasons #10581

See #10531, #10538 for more background.

oxidecomputer/steno#462 fixes a Steno issue causing some sagas to erroneously fail to be recovered. However, there's an operational risk to fixing this, which is that it may cause ancient sagas that were failing to be recovered to start being successfully recovered. But these almost certainly can't be successfully run by current Nexus, since sagas are assumed to be tightly coupled to the specific Nexus version in which they were created. Fixing this issue (i.e., having Nexus explicitly abandon sagas that it fails to recover for non-transient reasons) eliminates that operational risk.

(You could argue that it's fixing the wrong problem: the real problem is that Nexus should never try to run a saga from a different version of itself. That's true, and we considered that as well on today's update sync, but that's complicated and carries its own risks. The change proposed here appears much simpler and safer and addresses the immediate problem.)

Fixing this issue also fixes the health check that #10538 seeks to fix. By having Nexus abandon sagas that it cannot recover, the previous health check behavior that looked for long-running sagas in the database will work unmodified.

Another argument for this is: if Nexus fails to recover a saga for non-transient reasons, there's no reason to think recovery will ever work again. Only a software change (like steno#462) could resolve that, but a software change necessarily means the saga shouldn't be run. So we may as well abandon it.

Finally: this does create a new problem, which is abandoned sagas. See RFD 555 for more on this. These generally don't create major problems for the control plane, but they do potentially tie up resources and leave user-facing objects (like instances and disks) in strange intermediate states. What mitigates this is that the blast radius is generally limited to the affected resource. The plan here should be that we elevate these to the customer and then support (via #10530) and then provide remediation (which is necessarily saga-specific).