Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory error compressing to zip using zstd with level 20+ #388

Open
pps83 opened this issue Jan 11, 2025 · 6 comments
Open

Out of memory error compressing to zip using zstd with level 20+ #388

pps83 opened this issue Jan 11, 2025 · 6 comments

Comments

@pps83
Copy link

pps83 commented Jan 11, 2025

I use 7-Zip ZS 22.01 - v1.5.5 - Release 3 that seem to be very outdated compared to the src code. I wanted to do some tests with .zip + zstd and simply wanted to compress src code of the latest lzma sdk. If I use .zip+zstd with compression levels above 19 I get "The system cannot allocate the required amount of memory" error:

image

these are the compression settings:

image

7z+zstd with 20+ compression level has no issues, so it's some sort of bug in the code. Total uncompressed size of the code is less than 10MB.

@sebres
Copy link
Collaborator

sebres commented Jan 13, 2025

Can confirm it with current master too, however only with level 22 (seems to be depending on max memory).

Attempts to reduce Memory usage for Compressing (no matter to which value), in order to workaround it, cause another error - "The parameter is incorrect".

@inschrift-spruch-raum
Copy link

Attempts to reduce Memory usage for Compressing (no matter to which value), in order to workaround it, cause another error - "The parameter is incorrect".

I have also noticed this issue. On my computer, no matter how 7z is set up, this error occurs, while other formats do not have this problem

@mcmilk
Copy link
Owner

mcmilk commented Feb 24, 2025

Hello,
normally the error just says that there is no free memory left for doing the thing.
The documentation of zstd says, that level > 20 really needs more memory then normal: https://facebook.github.io/zstd/zstd_manual.html#Chapter1

Can you check via e.g. wtime - that your system has enough memory?

Thanks, Tino

@pps83
Copy link
Author

pps83 commented Feb 24, 2025

you can try it yourself, there is a bug. I have more than enough and I was compressing a tiny file.
I use zstd in my code and I was testing if my result matches 7zip+zstd. Somehow zip+zstd with high levels simply fails

@mcmilk
Copy link
Owner

mcmilk commented Feb 24, 2025

Okay, the ZIP code in general seems to have some problem... so I think, I can fix this by fixing #391 ...

@pps83
Copy link
Author

pps83 commented Feb 24, 2025

I have my own code to read/write zip files and I added zstd compression to it. Just wanted to check if my code was correct and able to read/write these zip+zstd files. For that I tried to create a few test files and got the error.
Simply try to create ZIP file with ZSTD compression level=22. 7z+zstd has no issues, only the zip+zstd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants