Skip to content

Avoid reading the whole file into memory when adding a file to a .zip archive #7123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Dec 5, 2021

Conversation

gave92
Copy link
Member

@gave92 gave92 commented Nov 28, 2021

Resolved / Related Issues
Items resolved / related issues by this PR.

Details of Changes
Add details of changes here.

  • Avoid reading the whole file into memory when adding a file to a .zip archive

Unfortunately this requires some trivial changes to SharpZipLib which currently does not support opening a ZipOutputStream to an existing archive. Modified nuget package added to the project, like it was done for microsoft.management.infrastructure.runtime.win

To test: copy a 1GB and paste it into a .zip archive. See Files.exe memory not-growing anymore to 1GB :)

Validation
How did you test these changes?

  • Built and ran the app

@d2dyno1
Copy link
Member

d2dyno1 commented Nov 28, 2021

Does this fix zip bombs?

@gave92
Copy link
Member Author

gave92 commented Nov 28, 2021

My understanding is that zip bombs are about extracting archives, this reduces memory usage when compressing files.

@gave92
Copy link
Member Author

gave92 commented Nov 28, 2021

But now I'm curious to test what happens extracting the famous 42.zip in Files, let's see ;)

@gave92
Copy link
Member Author

gave92 commented Nov 28, 2021

Ah nevermind: those kinds of zip bombs only work if you try recursively unzip nested archives (which we don't). This one (non recursive) works though: https://www.bamsoftware.com/hacks/zipbomb/. It's a 42Kb file that gets expanded to 5Gb, but the only consequence is that you occupy some disk space (and you can stop extraction at any moment).

@gave92 gave92 marked this pull request as ready for review November 28, 2021 19:05
@gave92 gave92 changed the title [TEST] Avoid reading the whole file into memory when adding a file to a .zip archive Avoid reading the whole file into memory when adding a file to a .zip archive Nov 28, 2021
@yaira2 yaira2 requested a review from d2dyno1 November 28, 2021 19:11
@yaira2
Copy link
Member

yaira2 commented Nov 28, 2021

@gave92 would it make sense to switch to using 7zip?

@gave92
Copy link
Member Author

gave92 commented Nov 28, 2021

@yaichenbaum we can look at integrating 7zip through e.g SevenZipExtractor. I suggest we still merge this as it's beneficial for FTP too. Bundling a custom SharpZipLib is also not too bad if it's temporary (till we switch to 7zip).

@d2dyno1
Copy link
Member

d2dyno1 commented Nov 28, 2021

@yaichenbaum I'm going to review it next weekend. If you want to, you can review it now.

@yaira2 yaira2 self-requested a review November 28, 2021 19:53
Copy link
Member

@d2dyno1 d2dyno1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From initial look, it looks ok!

@yaira2 yaira2 added the ready to merge Pull requests that are approved and ready to merge label Dec 5, 2021
@yaira2 yaira2 merged commit 9d7062b into files-community:main Dec 5, 2021
@gave92 gave92 deleted the zip_memory branch December 5, 2021 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready to merge Pull requests that are approved and ready to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants