Repository navigation
MS Windows 32bit binary does not work properly for files bigger than 4 gigabyte #1911
Description
What version of ripgrep are you using?
ripgrep 13.0.0 (rev af6b6c5)
-SIMD -AVX (compiled)
How did you install ripgrep?
I use the MS Windows binary rg.exe from ripgrep-13.0.0-i686-pc-windows-msvc.zip
What operating system are you using ripgrep on?
MS Windows 10 32bit
Describe your bug.
For files bigger than 4 gigabyte result lines are missing.
Only result lines from the upper part of the file or no lines are printed.
What are the steps to reproduce the behavior?
Search e.g. in a 5 gigabyte file for content of the last line.
What is the actual behavior?
rg "string_in_last_line_of_big_file" bigfile.txt
no output
What is the expected behavior?
The last line of bigfile.txt containing "string_in_last_line_of_big_file" should be printed.
Activity
BurntSushi commentedon Jun 26, 2021
Does
--no-mmapresolve this?datatraveller1 commentedon Jun 26, 2021
Yes,
rg --no-mmap "string_in_last_line_of_big_file" bigfile.txtsolves the issue with the 32bit version!The missing result lines are printed with the --no-mmap option.
However, the previous 32bit version ripgrep-12.1.1-i686-pc-windows-msvc.zip works correctly without the --no-map option.
Some interesting hints about this observation:
ripgrep 12.1.1 (rev 7cb2113)
rg "string_in_last_line_of_big_file" bigfile.txt --debug=>
DEBUG|grep_regex::literal|crates\regex\src\literal.rs:58: literal prefixes detected: Literals { lits: [Complete(string_in_last_line_of_big_file)], limit_size: 250, limit_class: 10 }
DEBUG|globset|crates\globset\src\lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|grep_searcher::searcher::mmap|crates\searcher\src\searcher\mmap.rs:86: bigfile.txt: failed to open memory map: memory map length overflows usize
ripgrep 13.0.0 (rev af6b6c5)
rg "string_in_last_line_of_big_file" bigfile.txt --debug=>
DEBUG|grep_regex::literal|crates\regex\src\literal.rs:58: literal prefixes detected: Literals { lits: [Complete(string_in_last_line_of_big_file)], limit_size: 250, limit_class: 10 }
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: lz4: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: xz: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: brotli: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: zstd: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: uncompress: could not find executable in PATH
I somehow assume
--no-mmapshould be the default for the 32bit version to avoid wrong results.BurntSushi commentedon Jun 26, 2021
Yup. The next release will have mmap disabled in all non-64-bit systems.
I don't know why this used to work. The only semi-related change in ripgrep 13 that I can think of is that it is now statically linking vcruntime: #1613
I'm not sure why or how that would change things here, but it's simple enough to just disable mmap on 32-bit.
changelog: add #1911 bug fix
datatraveller1 commentedon Jun 27, 2021
Thank you very much! I'll add a few more comments just to ensure there are no misunderstandings with the fix:
I plan to use the MS Windows 32bit binary release both for my PC (32bit) and my laptop (64bit) like it used to work with version 12.1.1.
Like on Windows 32bit, the current 13.0.0 32bit binary release does not work correctly on Windows 64bit unless the --no-mmap option is used. Only the current 13.0.0 64bit binary release works correctly on Windows 64bit for big files.
So, the patch (disable mmap) should impact the target rust 32bit binary release and not the Windows version currently used by the user (32bit or 64bit). I'm not a Rust programmer but it seems to me the fix takes care of that.
BurntSushi commentedon Jun 27, 2021
Thank you for copying your comment from email. :-) I'll copy my response as well:
The patch I pushed should impact the target used to build the binary.
A
cfg!is conditional compilation.I don't use Windows, so it's a bit annoying to test this patch myself. I have a Windows laptop somewhere in my house,
but it's always a big production to pull it out because Windows
insists on spending hours running Windows Update.
If anyone has an easy way to build ripgrep's master branch for 32-bit Windows and test it on a >4GB file with a match at the end, that would be most appreciated!
adamreichold commentedon Sep 23, 2021
I seems it would suffice, but there is a small wrinkle to this for cross building for 32-bit Windows: Due to LLVM/Rust assuming SEH-style unwinding but most Linux distributions 32-bit MinGW toolchains using SjLj-style unwinding, one will run into linker errors related to
_Unwind_Resume, c.f. rust-lang/rust#12859 Some distributions like Fedora since version 32 however ship cross toolchains using SEH-style unwinding and using such a distribution seems to work as expected, c.f. #2000searcher: re-enable mmap on 32-bit architectures
textproc/ripgrep: update to 14.0.2
searcher: disable mmap searching on non-64 bit
textproc/ripgrep: update to 14.0.2
textproc/ripgrep: update to 14.0.2
textproc/ripgrep: update to 14.0.2