The performance gains on both speed and compression is night an day compared to zip. I made a simple test to try to compress a medium size project using both zip and zstd. Here are the result:
The compression speed was ~10x faster. The compression result was ~5% smaller.
Zstd
time gtar --zstd -cf monorepo.tar.zst monorepo
9.22s user 7.11s system 67% cpu 24.120 total
Zip
time zip --symlinks -q -r monorepo.zip monorepo
106.35s user 8.87s system 87% cpu 2:11.07 total
The uncompression speed was ~8 faster.
Zstd
time gtar --zstd -xf monorepo.tar.zst
3.52s user 15.62s system 117% cpu 16.348 total
Zip
time unzip -qq monorepo.zip
24.81s user 18.56s system 97% cpu 44.481 total
Installation and versions
# Builtin tar in OSX doesn't support zstd.
# Just provided OSX installation but pretty sure it's supported on very platform.
> brew install zstd gnu-tar
...
> zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
...
> gtar --version
tar (GNU tar) 1.35
...
> zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Zstd is maintained by Facebook. So it's a serious project.
https://github.com/facebook/zstd
AWS started using it and also saw good results.
https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/
Right now "Build teardown" is the slowest part of my whole pipeline and takes approx 3min. Majority of the time is compressing different artifacts using zip in serial.
If this was done both in parallell and using zstd this would probably take 10s instead of 180s.
My idea would be to support something like this in bitbucket-pipelines.yml
artifacts:
- type: zstd
pattern: some/file-pattern1/**
- type: zip
pattern: some/file-pattern2/**
- some/file-pattern3/** # Default to zip
