Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compress release bundle with zstandard/zstd to reduce size #2400

Open
DSmithVA opened this issue Jul 31, 2024 · 1 comment
Open

Compress release bundle with zstandard/zstd to reduce size #2400

DSmithVA opened this issue Jul 31, 2024 · 1 comment

Comments

@DSmithVA
Copy link

I propose a .zstd download option alongside the existing .gz one for Linux releases. For the latest 2.18.1 linux64 bundle, using zstd instead of gzip can cut off 33% of the file size, or 822.8 MiB down to 553.2 MiB.

Example command to convert the existing .gz:
zcat codeql-bundle-linux64.tar.gz | zstd --long=27 -9 -o codeql-bundle-linux64.tar.zstd

File sizes:
862823301 codeql-bundle-linux64.tar.gz
580124258 codeql-bundle-linux64.tar.zstd

For zstd arguments, compression levels above -9 saw diminishing returns, though -19 does get down to 504.5 MiB while taking 12x longer to compress. Using higher --long= values improves compression, but 27 is the highest value that clients can process by default, per https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md?plain=1#L162

Compression with xz is also an improvement, it's just noticeably slower. Either is an improvement over just .gz and any recent linux will support both .zstd or .xz for decompression.

@jketema
Copy link

jketema commented Jul 31, 2024

Thanks for your feedback. We'll take this into consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants