pipe scp: extremely slow performance

Question

hello, i'm been waiting for an hour now for 300mb to be transferred over to google vm. is this normal? Looks like i'm getting ~1MB a minute. And from home laptop, is getting 2MB/s upload speed to the same instance. Mon Jan 24 07:13:37 UTC 2022 az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/ 104M /tmp/test/node_modules/ az@instance-5:~/test/test123$ date Mon Jan 24 07:14:22 UTC 2022 az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/ 105M /tmp/test/node_modules/ bitbucket_pipelines.yml artifacts: - node_modules/** - .nuxt/** - package.json - .env - step: name: Deploy node_modules artifacts using SCP deployment: test script: - pipe: atlassian/scp-deploy:1.2.1 variables: USER : $DEP_USER SERVER : $DEP_SERVER REMOTE_PATH : '/tmp/test/' LOCAL_PATH : 'node_modules/' Logs: Build teardown 51s You already have a 'node' cache so we won't create it againSearching for files matching artifact pattern node_modules/**Artifact pattern node_modules/** matched 38418 files with a total size of 252.4 MiBCompressed files matching artifact pattern node_modules/** to 48.8 MiB in 34 secondsUploading artifact of 48.8 MiBdeploy: pipe: atlassian/scp-deploy:1.2.147s --volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \ ...+ docker container run \ INFO: Configuring ssh with default ssh key. INFO: Adding known hosts... INFO: Appending to ssh config file private key path INFO: Applied file permissions to ssh directory. Status: Downloaded newer image for bitbucketpipelines/scp-deploy:1.2.1

Norbert Csupka · Answer

Hi @Admin Admin ,

Thank you for your reply. I would like to inform you that yes, you're correct, the scp pipes image is approximately 1GB:

bitbucketpipelines/scp-deploy 1.2.1 819d480f462b 3 months ago 927MB

The reason behind this is because this pipes is based on python:3.8 image, which is 808MB:

python 3.8 7684fd0fc298 4 days ago 885MB

I would like to inform you that SCP is not capable to check CRC, thus we also don't do it. SCP also doesn't support parallel transfers

As you're copying more than 38000 small files, this is a normal behavior that it takes a while to copy that directory.

As a workaround I can recommend you the following:

* Use the RSYNC pipes as it's more suitable to copy several small files
* In case you wish to keep using SCP pipes, I can highly recommend you to compress those small files and copy the compressed file from Pipelines to your server.

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support

Admin Admin_zz7tsr · Answer

scp -> sftp: i doubt it matters.

BUT I'll try to get that done time permits.

Admin Admin_zz7tsr · Answer

Norbert,

i'm using the premier tier (https://cloud.google.com/network-tiers/) and the VM is in us-central1-a.

The transfer for the _single_ large file is significantly faster.

I sense the difference in performance is due to roundtrip time for each small transfer.. and maybe you are checking the CRC for the delivered file?

Maybe you can introduce a logic for parallel transfer or expose these params to the user to play with?

In addition:

I noticed that the scp pipe's image is ~1GB? is this a correct observation?