Add a pure-Rust backend

[brson]

flate2 gets its actual compression from either libz or miniz, both C libraries. Ultimately, a pure-Rust stack is better for Rust, mostly because of build simplicity.

Write a miniz replacement in Rust, publish it to crates.io and add another compile-time feature to select it.

Then make the performance better than miniz and make it the default.

[brson]

There are existing 'deflate' and 'inflate' crates that might be suitable.

[oyvindln]

Developer of the deflate crate here, I would be happy to coordinate with you on this if you decide to utilize it.

[dtolnay]

Notes from today's libs team meeting:

• Adding a pure-Rust backend should not require a change to the API.
• Opting into C backends can cause mysterious explosions when multiple crates opt in to multiple revs.
• Not a great risk here because miniz is not going to change.
• A pure-Rust backend is not a requirement for 1.0 of flate2.

Currently we do not plan to pursue a pure-Rust backend for flate2. Instead we plan to promote migration to pure Rust crates as they become stable.

[matklad]

Currently we do not plan to pursue a pure-Rust backend for flate2. Instead we plan to promote migration to pure Rust crates as they become stable.

Would a drop-in pure rust replacement of miniz be useful anyway? We have summer internships at https://compscicenter.ru, and porting a C library to Rust seems like an interesting project (given that you can throw corrode, ALF and all other cool tools at it) which I probably can mentor.

[alexcrichton]

I'd imagine so yeah! ABI compatible, yet safe, libraries seem like a fantastic use case for Rust.

[Frommi]

I'm a summer intern mentored by @matklad to rewrite miniz to Rust. Here is the repo.

[Frommi]

I made a fork that uses miniz_oxide. Right now miniz.c file is fully ported to Rust and a little bit from tdef.

[Frommi]

Deflation is done (except for png functions), but it certainly can be a bit rustier. As a whole miniz_tester.cpp is working 1.09 (SE 0.01) times slower with miniz_oxide according to /usr/bin/time user time.

[oyvindln]

The performance of miniz_oxide for deflating looks really good!
I tested it with the benchmarks I use to compare my own encoder with flate2 and the difference between the oxide and the vanilla version of flate2 is pretty small (except for creating a new writer instance):

flate2-oxide
        Running target/release/deps/bench-320f073631836af1

running 9 tests
test test_file_zlib_def         ... bench:  10,611,928 ns/iter (+/- 215,276)
test test_file_zlib_fast        ... bench:   2,806,482 ns/iter (+/- 54,799)
test test_file_zlib_flate2_best ... bench:  13,205,205 ns/iter (+/- 411,780)
test test_file_zlib_flate2_def  ... bench:  10,080,656 ns/iter (+/- 231,622)
test test_file_zlib_flate2_fast ... bench:   2,620,335 ns/iter (+/- 73,291)
test test_file_zlib_high        ... bench:  13,603,453 ns/iter (+/- 150,140)
test test_file_zlib_rle         ... bench:   1,189,553 ns/iter (+/- 19,496)
test writer_create              ... bench:       6,852 ns/iter (+/- 251)
test writer_create_flate2       ... bench:      14,419 ns/iter (+/- 385)

flate2-miniz
     Running target/release/deps/bench-320f073631836af1

running 9 tests
test test_file_zlib_def         ... bench:  10,627,654 ns/iter (+/- 138,669)
test test_file_zlib_fast        ... bench:   2,799,167 ns/iter (+/- 42,938)
test test_file_zlib_flate2_best ... bench:  12,888,815 ns/iter (+/- 609,888)
test test_file_zlib_flate2_def  ... bench:   9,629,206 ns/iter (+/- 104,458)
test test_file_zlib_flate2_fast ... bench:   2,033,945 ns/iter (+/- 45,261)
test test_file_zlib_high        ... bench:  13,614,552 ns/iter (+/- 176,195)
test test_file_zlib_rle         ... bench:   1,196,375 ns/iter (+/- 19,859)
test writer_create              ... bench:       6,823 ns/iter (+/- 211)
test writer_create_flate2       ... bench:       4,143 ns/iter (+/- 86)

(The test prefixed with flate2 is the ones using flate2)

Let me know if you want me to help out on making it more rusty. I suppose porting the miniz backend for flate2 may be an easier route to get rid of this dependency on a C-library than trying to approach the speed with the current pure-rust decoders.

[Frommi]

writer_create_flate2 is slower probably because I 0-set all the arrays at init. Valgrind is actually reporting conditional jumps based on uninitialized values in miniz because of that.

[oyvindln]

Yeah it's probably 0-initialization taking a little bit of time.

I believe the reading of uninitialized values is because it may read past the end of the input buffer when comparing matches as it doesn't check if it's hit the end. Zlib does something like that deliberately to avoid doing too many bounds checks when comparing matches.

[alexcrichton]

Nice work @Frommi! Those are some really encouraging numbers :)

[Frommi]

With help from @oyvindln inflation is now rewritten as well. miniz_oxide should already be usable, but there is still some work to be done like comprehensive fuzzing, testing for ARMs and some optimizations.