zpool commands block when a disk goes missing / pool suspends

[gordan-bobic]

It would appear that pulling a disk from a single disk pool causes ZoL to get into a state where all zpool commands (e.g. zpool list) to block. sync also blocks indefinitely. Both become uninterruptable (kill -9 doesn't work).

No other errors in dmesg other than the disk getting disconnected (I removed it) and:

WARNING: Pool 'poolname' has encountered an uncorrectable I/O failure and has been suspended.

There need to be timeouts and the failure handling of this scenario needs to be more graceful than requiring reboot of the machine.

[DeHackEd]

This isn't a case of a time out. ZFS knows the disk is gone. This is a deliberate choice by the ZFS developers. If your pool can't survive due to redundancy failures it enters a suspended state to allow the administrator the ability to fix it while dirty data, etc are still in RAM. zpool set failmode=continue poolname allows some operations to fail with IO errors rather than jumping to suspended mode ASAP, but some actions will still suspend the pool.

What might make sense is modifying some zpool tools to be aware of suspended pools and avoid querying data that may require disk IO to find.

Here's a sample pool I faulted using blkdiscard on an SSD with failmode=continue

  pool: testpool
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-JQ
  scan: none requested
config:
    NAME        STATE     READ WRITE CKSUM
    testpool    UNAVAIL      0     0     4  insufficient replicas
      vdb1      UNAVAIL      0     0    16  corrupted data
# zpool list
NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
testpool  49.8G  10.1M  49.7G         -     0%     0%  1.00x  UNAVAIL  -

[gordan-bobic]

So why do sync and zpool list also hang forever? There are other pools in the machine, and it is a zfs-only machine. Hanging until the machine is rebooted just seems downright silly. For this behaviour to make any sense there has to be a way to either tell ZFS to give up on a pool and/or unsuspend the pool.

[DeHackEd]

This is intended behaviour to safeguard your dirty data and allow an administrator the chance to fix a broken machine while hot data is still available. The sync can't complete because there is insufficient disks to write out the transaction. zpool list hanging could be interpreted as a bug - unknown field values could be replaced with just a - dash and only display pool name and health (FAULTED, UNAVAIL, etc).

If you can fix the problem, ZFS will pick up where it left off and unsuspend.

[gordan-bobic]

Except that when it all hangs it is not possible at that point to
zpool set failmode=continue poolname

There really needs to be a way for the administrator to say "fine, go fail" without requiring a complete reboot - especially when that reboot actually requires a hard-reset because the shutdown sequence will also hang when trying to cleanly export the pools.

[behlendorf]

@gordan-bobic the situation should be improved somewhat in the 0.6.4.1 tag. At a minimum command which do not require disk IO should be allowed to complete.

For example, commands like zpool list which should not have been impacted but were because parts of the output required accessing the disk. The very disk which was no longer available. Specifically this was a regression which crept in with feature flag support, see 417104b for details. However, command list zfs list potentially won't because some dataset needs may need to be read from disk.

The fact that you can't always CTRL-C the command does definitely sound like a bug.

[GregorKopka]

@behlendorf imho that zfs list could block is a bug: it is read-only, and fundamental to zfs.

Data that is needed for zfs list should be always be in ARC and stay there till the pool is exported.

Ideal would be that the metadata needed for this is loaded on pool import and never ever be released (at least an option to configure such behaviour should be available).

More on topic: Block operations that require r/w access to a suspended pool (apart from listing it) seems reasonable, nevertheless they should always be abortable by a signal and clean up correctly so that they don't leave dangling locks. Blocking operations on healthy pools just because another pool in the system failed and/or reaching a system state that can only be cured by a reboot is imho a bug,so there should be a way to cleanly (and completely, so a reimport would be possible) remove a suspended pool from the system.

[behlendorf]

@GregorKopka regarding zfs list that's potentially a lot of data. There could be 100,000's of datasets when you start including snapshots. Plus you'd need to store the properties for all of them, it has the potential to consume a significant chunk of memory. I could see this potentially being a configurable thing.

On the other points I generally agree. But the devils always in the details for these things so someone will need to investigate why it is the way it is.

[jerome-diver]

i confirm, when an usb device miss (due to hard un-connection), services freeze, zfs and zpool are no more been usable (from antergos zfs packages, but i think also from others), and CTRL+C or CTRL+Z not respond. Ugly... need to reboot... at this time and because of this, ZFS is not a stable/strong file system format and could not safety be use with external hard drive.

[mailinglists35]

duplicate of #3256 ?

[jerome-diver]

not really, this subject is specific to 'disk removed (without to be wanted for)'.
ZFS dev team need to have consideration about material reality that can failed or be removed on this real factual world, that's real situations that does happened by the way to everybody. Close it and target on the other subject is like to try to not see the issue origin.
A disk that has been unconnected, has not to be a crucial problem that can make all your data loosed, because the world is not perfect and ZFS has to works in this real world (and sure... it is not so simple to look on this way because reality is not theorical things). Also, many other file system take care about this reality.
i think this post would be close when ZFS would be able to be stable and strong on this real world situations, and specifically when a drive has been hard unconnected.

[behlendorf]

@jerome-diver you can set the failmode=continue zpool property to prevent the pool from suspending when the drive is hard removed from a non-redundant configuration. This will result in errors to the applications but it should not hang the system. This behavior is similar to what you'd get from other filesystems.

There is some related work under way to better be able to detect when a drive was removed and if it's readded to the system what the new device name is.