bzip2: merge into Go standard library

[darren]

it hasn't been updated for a while, is it stable yet?
if it is, hope it can merged upstream and fix golang/go#4828

[dsnet]

Hi, the version of bzip2.Writer in the development branch is very stable has gone through many months of fuzzing comparing it's implementation with that of the C version. I have not merged the development branch into master yet, but plan to do so in the near future.

In regards to golang/go#4828, the main roadblock is finding someone to be a suitable reviewer for the implementation. I'm not aware of anyone else who has a deep knowledge of how bzip2 works. @mdempsky and @nigeltao have reviewed a number of my compression related CLs (showing knowledge in LZ77 and Huffman encoding), but I don't know if they're knowledgeable about BWT. It also depends on the availability of free time on my behalf and those of the reviewers.

[nigeltao]

Happy to learn about BWT, but I'm really low on free time for the forseeable future. Sorry.

[mkaczanowski]

Hey, @dsnet any progress with merging bzip2 compressor into standard library? What help do you need to make it or what are the blockers?

[dsnet]

I am not currently working on merging it into the standard library. I would love to see that happen in the near future, but I am still blocked on finding appropriate reviewers.

The master branch of this repo has a stable implementation that has gone through CPU-years of fuzz testing.

[retnuh]

As an outside observer (i.e. someone who would like to use the library), does this really need a reviewer with deep knowledge of compressors? As a user of the library, all I really care about is that the outputs are compatible with other standard bzip2 implementations out there, and that it doesn't do anything crazy. Wouldn't a test suite would cover that?

Surely it's more important that there is a decent implementation available in the standard library now, rather than a potential future "best" implementation?

[dsnet]

outputs are compatible with other standard bzip2 implementations out there

Review by a another person doesn't guarantee this, but history has shown that programmers are faulty and the number of extra eyeballs tend to decrease the occurrence of bugs.

Wouldn't a test suite would cover that?

History has also demonstrated that test suites are often sub-par at detecting correctness bugs.

[dmitshur]

Surely it's more important that there is a decent implementation available in the standard library now, rather than a potential future "best" implementation?

If this package has the needed functionality, why is its name or location important? If API stability is important, it can/should be vendored.

i.e. someone who would like to use the library

Is there something that's preventing the package from being used at its current location?

[retnuh]

@shurcooL yes some people want only things in the base go distribution. The issue is about including/merging bzip2 in the standard library - see the original issue description.

@dsnet sure I get that and am all in favor of reviews but there's an argument to be made that something that seems to be working quite well is better than nothing, right?

[darren]

@shurcooL

If this package has the needed functionality, why is its name or location important? If API stability is important, it can/should be vendored.

It seems that you're a big fan of the amazing vendoring feature of go, so can you please remove net/http, encoding, compress, image etc from the standard library? If somebody needs then, JUST vendoring them, so simple and easy! I just don't need them while keep the go tool distribution bloated on my disk, pretty annoying:

du -sch  go1.10.1
357M	total

Oh, yes there's go 1.0 compatibility promise, can you PLEASE do this in go2.0?