CoW filesystems support

[axet]

Issue

CoW filesystems with random and frequent small writes create a lot of fragmentation which slows down file IO operations significantly.

For example. Using fresh btrfs+hdd filesystem allow to read write to partition at 100-150 MB/s speed. But when torrent got downloaded torrent file wlil be significantly fragmented and final reading speed of such file would be around 4-5 MB/s.

Here is simple fragmentation benchmark test which does:

1. create 5GB file on memory
2. measuring file fragmentation before start (1 2 or 50 fragments, depend on run)
3. make 20000 random writes into that single file
4. measure file fragmentation again, it will be about 40000 fragments.

As you can see fragmentation does not slowdown reading speeds for memory file, only affecting hdd devices.

   W none: 5,61GiB 0:00:05 [ 999MiB/s] [=====================================================>] 100%            
/mnt/copy: 5 extents found
fragmenting file...
/mnt/copy: 39469 extents found
   R none: 5,61GiB 0:00:03 [1,66GiB/s] [=====================================================>] 100%            
none 3383ms 1781793901/sec 101% (6109163520) E:-24048

W none,nodatacow: 5,61GiB 0:00:05 [1,01GiB/s] [==============================================>] 100%            
/mnt/copy: 36 extents found
fragmenting file...
/mnt/copy: 35 extents found
R none,nodatacow: 5,61GiB 0:00:03 [1,67GiB/s] [==============================================>] 100%            
none,nodatacow 3361ms 1793456937/sec 100% (6027808768) E:0

   W none: 5,61GiB 0:00:05 [1006MiB/s] [=====================================================>] 100%            
/mnt/copy: 27 extents found
fragmenting file...
/mnt/copy: 39458 extents found
   R none: 5,61GiB 0:00:03 [1,60GiB/s] [=====================================================>] 100%            
none 3507ms 1718793489/sec 101% (6109151232) E:-23194

   W zstd: 5,61GiB 0:00:05 [ 971MiB/s] [=====================================================>] 100%            
/mnt/copy: 22 extents found
fragmenting file...
/mnt/copy: 39452 extents found
   R zstd: 5,61GiB 0:00:03 [1,57GiB/s] [=====================================================>] 100%            
zstd 3582ms 1682805351/sec 101% (6109118464) E:-22699

   W zstd: 5,61GiB 0:00:05 [ 961MiB/s] [=====================================================>] 100%            
/mnt/copy: 32 extents found
fragmenting file...
/mnt/copy: 39456 extents found
   R zstd: 5,61GiB 0:00:03 [1,57GiB/s] [=====================================================>] 100%            
zstd 3575ms 1686100354/sec 101% (6109134848) E:-22748

Real HDD test data. Initial read/writing speed 173MB, after 100000 write operations, 118586 fragments, drops reading speed to 6MB/s.

    setup: 1,00GiB 0:00:02 [ 433MiB/s] [=====================>] 100%            
   W none: 1,00GiB 0:00:05 [ 175MiB/s] [=====================>] 100%            
/media/axet/2TB/123//copy: 1 extent found
fragmenting file...
/media/axet/2TB/123//copy: 118586 extents found
   R none: 1,00GiB 0:02:50 [6,02MiB/s] [=====================>] 100%            

Here is a script:

• https://gitlab.com/axet/homebin/-/blob/debian/homebin.d/benchmarks/btrfs-benchmark-frag

Feature request

We need option for torrent client which mark all new pre-allocated files created with NO CoW flag (ala chattr +C). And remove such flag after torrent file fully loaded.

Workaround

Here is a workaround:

1. create btrfs subvolume named tmp (btrfs sub create tmp)
2. mark it no CoW (chattr +C tmp)
3. set torrent to use incomplete folder location at 'tmp'

Solution

Workaround is not optimal, since whole torrent got moved every time when rehashed to download portion of the torrent. Good implementation would move only torrent portion which is need to downloaded. It could be separate option for 'Incomplete Files folder' in addition to 'Incomplete Torrent folder'.

Option 'Incomplete Files folder' can be enabled automatically within hidden folder when app detects CoW filesystem. For example when I start to download 'BestMoive/{Movie.avi,README.txt} it would create BestMovie/.nocow folder using 'btrfs sub create .nocow' And then mark it using chattr +C .nocow. Then move only parts need to be downloaded and skip parts with zeros (not downloaed to avoid moving new pre allocated torrent files). Then finish download. Then move new data back to its original location. Then delete '.nocow' folder

[sysdump]

Is this with "Pre-allocate disk space for all files" on or off?

[axet]

It is on. But surely it does not matter if you look at test data.

[axet]

@arvidn Could be part of qbittorrent or core library it self. Happy new year!

[arvidn]

@axet My udnerstanding is that CoW filesystems would only actually perform a copy-on-write if there are more than 1 reference to the inode. It doesn't make much sense to make a copy, mutate it and then discard the previous inode.

Could you point to some documentation of how btrfs behaves with regards to copy on write?

There may be other reasons to trigger a copy-on-write. For example if you have a back-up system take a snapshot of the whole filesystem or the file in question. However, I would only expect such snapshots to exist while performing the back-up/copy. If the file (being downloaded) is written to at the same time, I would expect a single copy to be made. Any subsequent writes would go to the newly created copy, with just a single reference to it. i.e. no second copy.

In your benchmark, you don't include a step where you pre-allocate the space for the file. Are you sure that's not what's causing the fragmentation in your test?

[arvidn]

it appears btrfs is unique in that it makes all writes atomic by employing CoW. I believe my description above describes ZFS and APFS.

[axet]

My benchmark script creating fresh btrfs filesystem it has no snapshots. It has single file created using /dev/random and second file copy of it. I do not know how btrfs works, clearly this is default behavior.

Only difference if I preallocate file would be 1 fragment instead of 25 initial frragments. But should not affect further fragmentation during sequential random writes. Easy to test, but I do not think this is necessary.

[arvidn]

arvidn/libtorrent#8103

[arvidn]

@axet is it possible for you to test the above PR?

[axet]

Sure I can test a patch.

But this implementation looks very limited. NoCoW flag can only be set for zero length files, not 100% sure, but I do not see file size checks before calling ioctl. And after NoCoW flag set, it is impossible to remove such flag, unless file is fully duplicated and all data moved to new file without NoCoW flag. It can be simplified using btrfs subvolumes, as described above or manually by actually copy whole data. Simply removing NoCoW will fail.

Also worth mention: NoCoW flag means no CRC, no compression, so it is not recommend to keep all files like that except for: databases, logs or torrents (downloading).

PS: Microsoft Copilot is completely insane.

[arvidn]

But this implementation looks very limited. NoCoW flag can only be set for zero length files, not 100% sure, but I do not see file size checks before calling ioctl.

It's called when the truncate flag is set, which means we're creating the file for the first time. And it's called before the file is pre-allocated, so it should be empty. It's also best-effort; any failure is silently ignored.

And after NoCoW flag set, it is impossible to remove such flag, unless file is fully duplicated and all data moved to new file without NoCoW flag. It can be simplified using btrfs subvolumes, as described above or manually by actually copy whole data. Simply removing NoCoW will fail.

I see. I will just remove the commented-out code then, and leave the flag set. Do you think that causes more problems than it solves? should disable_copy_on_write default to false?

[axet]

It is partial implementation. Not ready to be default. Here are more use cases:

1. New torrent / files need to set No CoW
2. Already allocated torrents need to move only actual data (without empty allocated space) to No CoW file. For example when torrent got rehashed.
3. After file downloaded need to move file data to remove CoW flag by moving data.

It can be implemented as a function which knows how to move only data (skipping allocated holes without data). For example when we have allocated 100GB torrent and only 1GB data is downloaded or rehashed before torrent started partially downloaded files has to be moved to No CoW file. Then when file fully downloaded moved back to CoW enabled. Same as 'Append !qB extension to incomplete files' but for CoW files.

[arvidn]

1. New torrent / files need to set No CoW

This is covered by arvidn/libtorrent#8103, right?

2. Already allocated torrents need to move only actual data (without empty allocated space) to No CoW file. For example when torrent got rehashed.

The existing function to move or copy files already skip sparse regions. however, the new files don't get the NOCOW flag set. I don't think it's important to "fix" existing files. Users can do that manually if it's important to them. Searching the internet, it seems some users are already setting this flag on download folders and mounts.

3. After file downloaded need to move file data to remove CoW flag by moving data.

I'm thinking that this may be outside the scope of libtorrent. It seems like a really costly operation that users may need fine grained control over.