feature request: allow unloading a suspended pool from memory without exporting first

[mailinglists35]

when a pool is encountering i/o errors (because of a software bug or hardware issue/vdev gone missing), zol suspends the pool until it is brought to functional state.

most of the time this is impossible and user must hard reboot the server (hardware button or echo b > sysrq-trigger)

the forced reboot does not export the pool but simply decouples the zfs code from the storage devices.

please add this feature natively so I can "unload" a pool without commiting the export operation to disk, to further be able to unload the zfs modules and try the latest git master version (for example) without having to reboot the os.

[mailinglists35]

this can solve #4003 #2023 #2878 #3256

[fling-]

shutdown hangs for me even when a non-root pool gets suspended

[mailinglists35]

pinging @behlendorf

[mailinglists35]

note that in #3256 even if you bring back the device, the pool remains stuck in suspended state

[behlendorf]

If someone is interested in working on this by all means let me know.

[dweeezil]

I did a bit of research into un-suspending a pool. One thing that seems to cause problems is using volatile names for the vdevs such as /dev/sdX. If a pool is created as raidz sdb sdc sdf and is suspended by removing sdb and sdf, and then the bus is rescanned so they come back, the pool won't unsuspend following a zpool clear (and/or zpool reopen) if they come back as different device nodes. Using stable names such as from /dev/disk/by-id appears to work just fine.

I realize the feature being requested here is to "abandon" a suspended pool but I figured I'd point out the issue with the vdev names. It seems the biggest issue in being able to abandon a suspended pool is dealing with unmounting its filesystems and dealing with any processes which may have its zvols open.

[dweeezil]

I'm commeting here because this issue appears to be the best defined of the bunch of related issues, some of which have been referenced here earlier.

After some private correspondence with others, I've seen two cases arise: The first, and best defined, is a pool which has been "cleanly" suspended via zio_suspend(). The second, and more difficult to define, is a pool in which there were sufficient checksum errors and, typically not enough pool redundancy to correct them, that a VERIFY fails and panics one of ZFS' threads; typically the sync task.

As an example of the second case, a pool with no redundancy had a spacemap in which one of its blkptr's 3 ditto blocks [EDIT] weren't properly were improperly written by the storage substrate as zeroes. This caused checksum errors the next time it was read and ultimately resulted in a panic due to dmu_buf_hold_array(), called in dmu_write(), returning an EIO and panicking due to a failed VERIFY. The events leading up to the panic were:

• FM_EREPORT_ZFS_DATA
• 3 instances of FM_EREPORT_ZFS_CHECKSUM, one for each ditto block
• 3 more instance of FM_EREPORT_ZFS_CHECKSUM, one for each ditto block

That set of 4 events was repeated one more time prior to the panic.

The feature desired, as indicated in this issue, is to unload (I've been using the term "evacuate") a pool in either of these states to avoid rebooting. Presumably, the faulted pool is some sort of back-up or is otherwise not critical to the system's operation but it does continue to consume resources and can also cause processes to block, etc.

@behlendorf I plan on looking into this, beginning with what seems to be the more straightforward case of a pool that's simply been suspended. The second case is likelier to be trickier and, in thinking about it, if the system thinks a pool is toasted that bad (and in the case I described, the pool was toast, although likely mostly recoverable with some careful gymnastics), it would argue for a non-panicking mode for non-debug builds in which the pool were put in some other type of suspended state in which the only option would be to evacuate it.

[behlendorf]

@dweeezil that's great. The suspended pool case is definitely something which can be handled better and should be relatively straightforward. It should largely be a matter of tearing down everything cleanly and returning errors to any active ZFS consumers.

As for the second case if a VERIFY or ASSERT is hit that is by design intended to be fatal and unrecoverable. For cases where it's possible to hit a VERIFY due to an IO error we're going to need to replace that VERIFY with proper error handling code. So for example dmu_write() will need to be updated to return an error code and all of it's callers updated to handle it correctly.

[mailinglists35]

I did a bit of research into un-suspending a pool

@dweeezil is there any hope for a functional PR in a not so distant future? :-)
especially for the second case you described :D

[dweeezil]

@mailinglists35 Sorry, but this issue seems to have resisted percolating sufficiently high on my to-do list to get the attention it deserves. Are you mainly interested in the case where a VERIFY is hit and causes a panic? Your original report didn't have any zpool status or syslog output. I will try to give this and the referenced issues a fresh look in the next couple of days.

[mailinglists35]

@dweeezil Are you mainly interested in the case where a VERIFY is hit and causes a panic? Your original report didn't have any zpool status or syslog output

sorry, I assumed referencing an example issue in the first comment after filing the issue should be enough.
I am interested in all the cases where zpool export is unable to return success, regardless of the cause, so the desired result is to be able to exit the situation (forcibly "drop" the pool in the same way as if I pressed the hardware reset button) without a reboot

Here is a relevant one from those enumerated initially, containing zpool status and dmesg

#3256

also (no logs, just status) #3461

[mailinglists35]

I think this comment summarizes in a clearer language the feature request:

"@gordan-bobic commented on Dec 15, 2016
There really needs to be a way to instruct ZFS to throw away any and all dirty data and forget that the pool was ever here without rebooting the machine. Leaving a pool in a hung state with the disk removed is of no practical use. If there is risk of trashing the pool, so be it, but that risk doesn't seem any different from what happens if you reboot the machine, which is currently the only option anyway."

#3461 (comment)