pipe scp: extremely slow performance

[Admin Admin]

hello,

 i'm been waiting for an hour now for 300mb to be transferred over to google vm.

is this normal? Looks like i'm getting ~1MB a minute.

And from home laptop, is getting 2MB/s upload speed to the same instance.

Mon Jan 24 07:13:37 UTC 2022
az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/
104M /tmp/test/node_modules/

az@instance-5:~/test/test123$ date
Mon Jan 24 07:14:22 UTC 2022
az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/
105M /tmp/test/node_modules/

bitbucket_pipelines.yml

    artifacts:
      - node_modules/**
      - .nuxt/**
      - package.json
      - .env
- step:
    name: Deploy node_modules artifacts using SCP
    deployment: test
    script:
       - pipe: atlassian/scp-deploy:1.2.1
         variables:
           USER : $DEP_USER
           SERVER : $DEP_SERVER
           REMOTE_PATH : '/tmp/test/'
           LOCAL_PATH : 'node_modules/'

Logs:

Build teardown                                             51s

You already have a 'node' cache so we won't create it again

Searching for files matching artifact pattern node_modules/**

Artifact pattern node_modules/** matched 38418 files with a total size of 252.4 MiB

Compressed files matching artifact pattern node_modules/** to 48.8 MiB in 34 seconds

Uploading artifact of 48.8 MiB

deploy:

pipe: atlassian/scp-deploy:1.2.147s

+ docker container run \

   --volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \

...

Status: Downloaded newer image for bitbucketpipelines/scp-deploy:1.2.1

INFO: Configuring ssh with default ssh key.

INFO: Adding known hosts...

INFO: Appending to ssh config file private key path

INFO: Applied file permissions to ssh directory.

[Norbert C]

Hi there,

I'm Norbert from the other Community Ticket.

This is sounds interesting, can you verify what speed is your server having where you're trying to push the data? Does it have a 100mbps or 1gbps connection? Also can you let us know where is your located? Our Pipelines server are based in the USA. Does your server have a good bandwidth to US? 

Also for testing purpose, can you try to switch from SCP to SFTP and let us know if it makes the speed a bit faster?

I'm looking forward to hear from you.

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support

[Admin Admin]

Norbert,

 i'm using the premier tier (https://cloud.google.com/network-tiers/) and the VM is in us-central1-a.

The transfer for the _single_ large file is significantly faster.

I sense the difference in performance is due to roundtrip time for each small transfer.. and maybe you are checking the CRC for the delivered file?

Maybe you can introduce a logic for parallel transfer or expose these params to the user to play with?

In addition:

 I noticed that the scp pipe's image is ~1GB? is this a correct observation?

[Admin Admin]

scp -> sftp: i doubt it matters.

  BUT I'll try to get that done time permits.

[Norbert C]

Hi @Admin Admin ,

Thank you for your reply. I would like to inform you that yes, you're correct, the scp pipes image is approximately 1GB:

bitbucketpipelines/scp-deploy 1.2.1 819d480f462b 3 months ago 927MB

The reason behind this is because this pipes is based on python:3.8 image, which is 808MB:

python 3.8 7684fd0fc298 4 days ago 885MB

I would like to inform you that SCP is not capable to check CRC, thus we also don't do it. SCP also doesn't support parallel transfers

As you're copying more than 38000 small files, this is a normal behavior that it takes a while to copy that directory.

As a workaround I can recommend you the following:

• Use the RSYNC pipes as it's more suitable to copy several small files
• In case you wish to keep using SCP pipes, I can highly recommend you to compress those small files and copy the compressed file from Pipelines to your server.

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support

[Admin Admin]

re-run the test today:

date && du -sh ./node_modules/

Mon Jan 24 16:33:24 UTC 2022
1.1M ./node_modules/

date && du -sh ./node_modules/


Mon Jan 24 16:34:24 UTC 2022

2.4M ./node_modules/

same 1MB per minute...