Skip to content

tail: improve throughput for -c N file#11979

Merged
sylvestre merged 1 commit into
uutils:mainfrom
oech3:tail-non-plus
May 20, 2026
Merged

tail: improve throughput for -c N file#11979
sylvestre merged 1 commit into
uutils:mainfrom
oech3:tail-non-plus

Conversation

@oech3

@oech3 oech3 commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Part of #11607 (non-stream)
The code path without + is different.

$ target/debug/tail -c 1PB /tmp/huge | pv>/dev/null
^C.9GiB 0:00:01 [39.9GiB/s]
$ tail -c 1PB /tmp/huge | pv>/dev/null
^C02GiB 0:00:01 [2.02GiB/s]

@oech3

oech3 commented Apr 24, 2026

Copy link
Copy Markdown
Contributor Author
$ truncate -s 48k /tmp/page
$ taskset -c 0 hyperfine -N "target/release/tail -c 48k /tmp/page" "target/release/tail-splice -c 48k /tmp/page"
Benchmark 1: target/release/tail -c 48k /tmp/page
  Time (mean ± σ):       1.4 ms ±   0.2 ms    [User: 0.5 ms, System: 0.8 ms]
  Range (min … max):     1.1 ms …   4.0 ms    1836 runs
 
Benchmark 2: target/release/tail-splice -c 48k /tmp/page
  Time (mean ± σ):       1.3 ms ±   0.2 ms    [User: 0.5 ms, System: 0.7 ms]
  Range (min … max):     1.1 ms …   4.6 ms    1660 runs
 
Summary
  target/release/tail-splice -c 48k /tmp/page ran
    1.01 ± 0.25 times faster than target/release/tail -c 48k /tmp/page

Less than the size is very subtle. 32kib sometimes wins and loses.

@github-actions

github-actions Bot commented Apr 24, 2026

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/misc/io-errors is no longer failing!

@codspeed-hq

codspeed-hq Bot commented Apr 24, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 311 untouched benchmarks
⏩ 46 skipped benchmarks1


Comparing oech3:tail-non-plus (c14925f) with main (964610c)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@oech3 oech3 force-pushed the tail-non-plus branch 2 times, most recently from f95a2a1 to d6fbfa0 Compare April 24, 2026 16:21
@oech3 oech3 marked this pull request as ready for review April 24, 2026 16:35
@xtqqczze

Copy link
Copy Markdown
Contributor
1.01 ± 0.25 times faster than target/release/tail -c 48k /tmp/page

It's not statistically significant?

@sylvestre

Copy link
Copy Markdown
Contributor

is it interesting ? :)

@oech3

oech3 commented Apr 26, 2026

Copy link
Copy Markdown
Contributor Author

Someone might try to truncate header part from large iso file to extract a small file.

@xtqqczze

Copy link
Copy Markdown
Contributor

std::io::copy is already optimized to use splice(2) syscalls on Linux. If there’s a measurable performance gap here, it would likely be better addressed upstream in Rust’s standard library.

@oech3

oech3 commented Apr 26, 2026 via email

Copy link
Copy Markdown
Contributor Author

@oech3

oech3 commented Apr 26, 2026 via email

Copy link
Copy Markdown
Contributor Author

@xtqqczze

Copy link
Copy Markdown
Contributor

More importantly, io::copy breaks some FUSE filesystem's stat. But upstream is clearly not interested to fix it. So relying on io::copy is really bad idea.

@oech3 The PR description and comments should clearly state the motivation for this change. It was not apparent that this was intended to address a correctness issue with FUSE.

@oech3

oech3 commented Apr 26, 2026 via email

Copy link
Copy Markdown
Contributor Author

@oech3

oech3 commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

If I am understanding correctly, we were not extending pipe size to 1 MiB at #6964 (comment) . But splice call was faster than io::copy at there already.

@oech3

oech3 commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

and rustix::pipe::fcntl_setpipe_size does not affect to io::copy :|

@oech3 oech3 force-pushed the tail-non-plus branch 2 times, most recently from aa27882 to 8962368 Compare April 30, 2026 06:07
@oech3

This comment was marked as resolved.

@sylvestre sylvestre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — nice throughput improvement using zero-copy splice for the bounded -c N path.

@sylvestre sylvestre merged commit 968f938 into uutils:main May 20, 2026
169 checks passed
@oech3 oech3 deleted the tail-non-plus branch May 20, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants