Does Bitbucket Pipeline has plans to support zstd for artifacts?

[Olle Bröms]

The performance gains on both speed and compression is night an day compared to zip. I made a simple test to try to compress a medium size project using both zip and zstd. Here are the result:

---

The compression speed was ~10x faster. The compression result was ~5% smaller.

Zstd

time gtar --zstd -cf monorepo.tar.zst monorepo
9.22s user 7.11s system 67% cpu 24.120 total

Zip

time zip --symlinks -q -r monorepo.zip monorepo
106.35s user 8.87s system 87% cpu 2:11.07 total

---

The uncompression speed was ~8 faster.

Zstd

time gtar --zstd -xf monorepo.tar.zst
3.52s user 15.62s system 117% cpu 16.348 total

Zip

time unzip -qq monorepo.zip
24.81s user 18.56s system 97% cpu 44.481 total

---

Installation and versions

# Builtin tar in OSX doesn't support zstd.
# Just provided OSX installation but pretty sure it's supported on very platform.
> brew install zstd gnu-tar
...

> zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
...

> gtar --version
tar (GNU tar) 1.35
...

> zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

Zstd is maintained by Facebook. So it's a serious project.
https://github.com/facebook/zstd

AWS started using it and also saw good results.
https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/

Right now "Build teardown" is the slowest part of my whole pipeline and takes approx 3min. Majority of the time is compressing different artifacts using zip in serial.

If this was done both in parallell and using zstd this would probably take 10s instead of 180s.

My idea would be to support something like this in bitbucket-pipelines.yml

artifacts:
  - type: zstd
    pattern: some/file-pattern1/**
  - type: zip
    pattern: some/file-pattern2/**
  - some/file-pattern3/** # Default to zip

[Syahrul · Accepted solution]

G'day, @Olle Bröms 

Welcome to the community!

Thank you for providing a detailed test and a concise explanation. At this moment, I don't have specific information regarding any planned changes to the artifact compression.

However, I have initiated to raise a feature request on your behalf. This will allow you to comment, vote directly, and stay updated. If we decide to explore your suggestion in the future, you will receive an update.

https://jira.atlassian.com/browse/BCLOUD-23151

Regards,
Syahrul