Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,460,392
Community Members
 
Community Events
176
Community Groups

pipe scp: extremely slow performance

Edited

hello,

 i'm been waiting for an hour now for 300mb to be transferred over to google vm.

is this normal? Looks like i'm getting ~1MB a minute.

And from home laptop, is getting 2MB/s upload speed to the same instance.

 

Mon Jan 24 07:13:37 UTC 2022
az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/
104M /tmp/test/node_modules/

az@instance-5:~/test/test123$ date
Mon Jan 24 07:14:22 UTC 2022
az@instance-5:~/test/test123$ du -sh /tmp/test/node_modules/
105M /tmp/test/node_modules/

 

bitbucket_pipelines.yml

    artifacts:
- node_modules/**
- .nuxt/**
- package.json
- .env
- step:
name: Deploy node_modules artifacts using SCP
deployment: test
script:
- pipe: atlassian/scp-deploy:1.2.1
variables:
USER : $DEP_USER
SERVER : $DEP_SERVER
REMOTE_PATH : '/tmp/test/'
LOCAL_PATH : 'node_modules/'

 

Logs:

 

Build teardown                                             51s
You already have a 'node' cache so we won't create it again
Searching for files matching artifact pattern node_modules/**
Artifact pattern node_modules/** matched 38418 files with a total size of 252.4 MiB
Compressed files matching artifact pattern node_modules/** to 48.8 MiB in 34 seconds
Uploading artifact of 48.8 MiB

deploy:

pipe: atlassian/scp-deploy:1.2.147s
+ docker container run \

--volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \

...
Status: Downloaded newer image for bitbucketpipelines/scp-deploy:1.2.1

INFO: Configuring ssh with default ssh key.

INFO: Adding known hosts...

INFO: Appending to ssh config file private key path

INFO: Applied file permissions to ssh directory.

2 answers

0 votes
Norbert C Atlassian Team Jan 28, 2022

Hi there,

I'm Norbert from the other Community Ticket.

This is sounds interesting, can you verify what speed is your server having where you're trying to push the data? Does it have a 100mbps or 1gbps connection? Also can you let us know where is your located? Our Pipelines server are based in the USA. Does your server have a good bandwidth to US? 

Also for testing purpose, can you try to switch from SCP to SFTP and let us know if it makes the speed a bit faster?

I'm looking forward to hear from you.

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support

Norbert,

 i'm using the premier tier (https://cloud.google.com/network-tiers/) and the VM is in us-central1-a.

 

The transfer for the _single_ large file is significantly faster.

I sense the difference in performance is due to roundtrip time for each small transfer.. and maybe you are checking the CRC for the delivered file?

Maybe you can introduce a logic for parallel transfer or expose these params to the user to play with?

 

In addition:

 I noticed that the scp pipe's image is ~1GB? is this a correct observation?

scp -> sftp: i doubt it matters.

  BUT I'll try to get that done time permits.

Norbert C Atlassian Team Feb 02, 2022

Hi @Admin Admin ,

Thank you for your reply. I would like to inform you that yes, you're correct, the scp pipes image is approximately 1GB:

bitbucketpipelines/scp-deploy 1.2.1 819d480f462b 3 months ago 927MB

The reason behind this is because this pipes is based on python:3.8 image, which is 808MB:

python 3.8 7684fd0fc298 4 days ago 885MB

I would like to inform you that SCP is not capable to check CRC, thus we also don't do it. SCP also doesn't support parallel transfers

As you're copying more than 38000 small files, this is a normal behavior that it takes a while to copy that directory.

As a workaround I can recommend you the following:

  • Use the RSYNC pipes as it's more suitable to copy several small files
  • In case you wish to keep using SCP pipes, I can highly recommend you to compress those small files and copy the compressed file from Pipelines to your server.

Best Regards,
Norbert
Atlassian Bitbucket Cloud Support

re-run the test today:

date && du -sh ./node_modules/

Mon Jan 24 16:33:24 UTC 2022
1.1M ./node_modules/

date && du -sh ./node_modules/


Mon Jan 24 16:34:24 UTC 2022

2.4M ./node_modules/

same 1MB per minute...

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS

Atlassian Community Events