Plan to migrate to zlib-ng

[carlossanlop]

For .NET 9, we are planning to replace our dependency of madler/zlib and zlib-intel with a dependency to zlib-ng.

The zlib-ng repo offers an alternative implementation of the zlib algorithm that:

• Is compatible and optimized for next-generation platforms and architectures.
• Adopted the known zlib-intel and Cloudflare optimizations.
• Deleted old and obsolete code.
• Adopted all known CVE fixes.
• Is actively maintained by multiple people and has a quick turnaround for fixing security issues. Additionally, zlib-intel is out of support already.

Acceptance criteria

Tests

• All tests must continue to pass, especially tests that read data.
• If needed, adjust tests that measure specific output characteristics of compression, but they must pass.
• Inputs which do not yield an Exception must continue to be Exception-free.
• Add unmanaged tests to directly verify the p/invoked methods.

Threat modeling

• Have a threat modeling exercise and make adjustments to the code if needed. Do this before collecting performance results.

Elapsed time

• Elapsed time for optimal compression should not increase more than 10%. "Optimal" should be deferred to the library's inherent notion of default. If the increase is higher than 10%, we should discuss if we need to choose a less agressive value manually to keep results aligned with the previous version.
• Elapsed time for fastest, smallest, and no-compression have no acceptance criteria, as they're entirely at the mercy of the underlying library. We may still collect and compare results.
• Elapsed time for optimal decompression should also be verified, but no acceptance target is defined, as very little managed code is involved in the decompression stage, and there are no algorithmic choices to be had. But the measurements can help us determine if we need to invest in zlib-ng's performance.

Output size

• Output size for optimal compression should not increase more than 10%, priority given to inclusive time. If the number differs due to the optimal compression being faster than baseline, then we need to decide if we should use a new manual value for optimal instead of deferring to the library default.

Managed memory

• Managed allocations should not increase when compared to madler/zlib or zlib-intel.

Unmanaged memory

• Peak unmanaged memory must be collected before and after. An acceptance criteria may be decided after comparing results, if needed.
• Test for unmanaged memory leaks. If any are found, fix locally, then contribute upstream if applicable.

Servicing plan

• No more than one release of .NET misses a given patch-release of zlib-ng. This takes in consideration the fact that zlib-ng does not have a predictable release cadence.
• .NET moves to a newer major or minor version of zlib-ng within 3 months of the first patch-release or 6 months of the initial release, whichever is earlier. This aligns with zlib-ng's history of maintaining parallel version support.
• zlib-ng takes security fixes as public PRs. There's a lag between accepting a fix and releasing it. We must be quick and reliably in taking patch commits, in addition to staying current on releases.
• Any team member can update to a newer version of zlib-ng in less than 10 minutes.
  • Must test and document the updating process prior to releasing .NET 9.
  • Decide what the quick process will be and implement it, for example, writing a script.

Execution plan

Milestone 1: Migration

• 1 day - Copy zlib-ng code to our repo: Migrate to zlib-ng, part 2: consume it in runtime #102403
• 2 weeks - Adapt zlib-ng to our repo: Migrate to zlib-ng, part 2: consume it in runtime #102403
  • 2 days - Understand our old cmake code and the new cmake code.
  • 2 days - Determine which flags are needed to meet our repo's criteria for building.
  • 1 week - Get the new cmake code to work Run additional CI legs in PR to test in all the supported configurations.
  • 1 day - Delete zlib and zlib-intel code: Migrate to zlib-ng, part 3: Remove zlib and zlib-intel source code and license mentions #104399
  • 3 days - Get the PR reviewed and iterate on feedback.
• 3 days - Threat modeling exercise. If any actions come out of the meeting, add them here and adjust estimates as needed.
• 5 days - Update xamarin/xamarin-macios and xamarin/xamarin-android. This can be done with a PR that is left open until those repos make the move to start consuming .NET 9, which is when the PR can be retargeted to their dependency flow PR.

Milestone 2: Testing

• 3 days - Run existing tests, ensure they pass.
• 1 day - Verify correct results in all supported architectures and platforms in our CI.

Milestone 3: Performance

• 2 weeks - Collect performance reports for before and after in Windows, Linux and MacOS, in x86, x64 and arm64 (Discuss if more configurations are needed). Compare results and discuss if action needs to be taken with the default values.
  • - Report for elapsed time. There should be no more than 10% change. [Perf] Linux/arm64: 35 Improvements on 7/9/2024 3:33:47 AM perf-autofiling-issues#38054
  • - Report for optimal size. There should be no more than 10% change

Milestone 4: Documentation

• 2 days - Collect a list of known first-party consumers of .NET compression APIs and notify them about this major change in .NET 9 - consumers of System.IO.Compression, System.IO.Packaging and OpenXML - Please update xamarin-android to consume zlib-ng from dotnet/runtime android#9082, Please update xamarin-macios to consume zlib-ng from dotnet/runtime macios#20811, cc @twsouthwick FYI
• 1 day - Find and update any mentions of zlib or zlib-intel in our dotnet repos, adapt to zlib-ng: Migrate to zlib-ng, part 3: Remove zlib and zlib-intel source code and license mentions #104399
• 1 day - Find and update any mentions of zlib or zlib-intel in dotnet-api-docs.
• 1 day - Document the zlib-ng update steps to ensure it's an easy process that anyone in the team can perform Instructions for adding a native external library dependency #105045

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

[jkotas]

Discuss if we want to add a compat switch to allow users to consume their own zlib DLL

I do not think we should be adding compat switches like this proactively.

[omajid]

Fedora moved recently from zlib to zlib-ng: https://fedoraproject.org/wiki/Changes/ZlibNGTransition

.NET (VMR) builds in Fedora are already using this (since it's the system zlib now) instead of the original zlib project. And we have used the self-built-VMR (release/8.0) that uses zlib-ng to build itself a few times now. We have not noticed any functional bugs/issues on Linux on arm64, ppc64le, s390x or x64.

[ericstj]

I do not think we should be adding compat switches like this proactively.

@jkotas we did this for ICU. We had a folks broken when we previously moved from managed implementation > zlib, and zlib > zlib-intel. I was imagining we'd need something as an escape hatch here. We wouldn't be carrying those alternative implementations but we could let folks bring their own. What worries you about providing a compat switch? Locking down the ABI between S.C.Compression and native?

[stephentoub]

I do not think we should be adding compat switches like this proactively.

@jkotas we did this for ICU. We had a folks broken when we previously moved from managed implementation > zlib, and zlib > zlib-intel. I was imagining we'd need something as an escape hatch here. We wouldn't be carrying those alternative implementations but we could let folks bring their own. What worries you about providing a compat switch? Locking down the ABI between S.C.Compression and native?

But there are actual and significant functional differences between both NLS and ICU and between ICU versions (with different data sets). I would not expect that to be the case with zlib; what specifically are we concerned about?

[carlossanlop]

what specifically are we concerned about?

If something stops working, something that we didn't catch in testing, we would have a workaround to give to customers: enable the compat switch, use your own DLL (it can be the old zlib DLL), while we fix the issue.

[stephentoub]

what specifically are we concerned about?

If something stops working, something that we didn't catch in testing, we would have a workaround to give to customers: enable the compat switch, use your own DLL (it can be the old zlib DLL), while we fix the issue.

That doesn't seem different from any other improvement we make. If we have blocking bugs, we service them.

[hopperpl]

Can we add support for compression/decompression via Span? This was always missing; zlib required using a stream.

This shouldn't be much of a problem as the native library already has exported functions for it.

[stephentoub]

Can we add support for compression/decompression via Span? This was always missing; zlib required using a stream.

This shouldn't be much of a problem as the native library already has exported functions for it.

This is unrelated to which zlib implementation is used. It is tracked by #39327.

[ericstj]

And #44793.

This issue is only about swapping the implementation of zlib.

But there are actual and significant functional differences between both NLS and ICU and between ICU versions (with different data sets). I would not expect that to be the case with zlib; what specifically are we concerned about?

Observable behavior. Size and content compressed payload. Maybe we'd want such a switch if we serviced this change in-place in 8.0 - but since we're doing in a major release we just accept this change as part of the churn.

[stephentoub]

since we're doing in a major release we just accept this change as part of the churn.

+1. We have such considerations any time we patch our existing zlibs, and we had one commensurate with the move to zlib-ng when we moved to zlib-intel.

[carlossanlop]

Thanks @jkotas @stephentoub and @ericstj for your input. Based on the feedback, I will remove the section for compat switch.

Maybe we'd want such a switch if we serviced this change in-place in 8.0

Speaking of which - I did not add a section to backport the zlib-ng migration to servicing branches. Do we need to consider this too?