Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Cloning Bitbucket repo into a pipeline

Hi,

 

I've seen soo many questions of this kind but any of them could solve my problem.

I have a repo A that pulls a repo B, both on Bitbucket. To make this work, I've created a SSH key for A and copy the its public key into Access keys on B.

 

Not sure if it matters but I've created my own packagist using satis (https://github.com/composer/satis).

 

From the composer logs, I can see this failing:

[401] https://bitbucket.org/foo/service/get/d70bca1785858d97e89aa553350ca9a19d797cd5.zip
Executing command (CWD): git config bitbucket.accesstoken
Failed to download my-vendor/service from dist: Could not authenticate against bitbucket.org
Now trying to download from source
- Syncing my-vendor/service (1.0.1) into cache
Cloning to cache at '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git clone --mirror 'https://bitbucket.org/foo/service.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git config bitbucket.accesstoken
No bitbucket authentication configured. Falling back to ssh.
Executing command (CWD): git clone --mirror 'git@bitbucket.org:foo/service.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git --version
Failed: [RuntimeException] Failed to execute git clone --mirror 'git@bitbucket.org:foo/service.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'

Cloning into bare repository '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git'...
Resource not found
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

I'm really out of options, can't see any source that helps me somehow.

 

thanks

1 answer

1 accepted

0 votes
Answer accepted

This looks like as if Satis or more likely Composer does not yet have these credentials and can't clone into its cache.

If this setup is within a pipeline that would puzzle me a bit honestly. Either you have a Satis setup and then Composer only needs to use it (that should then be without SSH as Satis uses HTTP/S) hence you should not see a git error.

However Satis here provides a source package then most likely and it does need the access then but default git SSH driver doesn't take it.

Are you able to clone directly with Git in the Pipeine already? At least for a test I would do it in front to more easily verify the SSH setup is correct. Satis/Composer may still fail, but you know the SSH configuration is good so far.

Next thing would be to check with Satis/Composer then and how to provide the SSH credentials so that Composer in the end can clone.

Some more Notes:

  • Composer has some automatism when seeing a Bibucket URL. It may kick in here so if this is news to you, verify with verbose output that Composer is doing the things you expect or not.
  • Composer has it's own auth/credentials mechanism, you may also need to add credentials to composer first, too (this might be especially true for source packages).

Hi @ktomk , thanks for your reply.

About the question you did, the answer is yes. I can clone these private repos directly. 

I'm aware of SSH credentials and I had to that to clone from Satis on the server I work on. Setting these credentials on my pipeline was one of my attempts but it didn't work too.

Based on what you said, I can say that there are others layers that are not allowing pipelines to clone from Satis. The only one I can think of in this moment is the firewall. The server I set the Satis is protected via firewall, so even with these credentials set, the firewall doesn't allow the request to go through. 

 

Please, let me know your thoughts about that.

Thanks, Felipe 

Hi Felipe, my question was if you can clone them in the pipeline. This should verify if transport is good or not before any Satis / Composer interaction.

The List of Public IPs from Bitbucket Pipelines, see here: https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/

Hi @ktomk

I'm sorry I wasn't clear. I said yes when I try to clone them in the pipeline, I'm able to do it. 

I'll have a look at this link you sent and give you a feedback.

 

thanks

Felipe

If cloning in the pipeline already works, the remote git repository is accessible via SSH.

The link then will not be that useful, as I put it in in case a firewall rule would have prevented SSH access to your remote server from the pipeline.

Now as SSH access works, and from the error message it is within composer install (right?), next thing to check might be composer.

The docs have a hint to SSH and private packages and authentication for private packages here:

From the second link, I've set bitbucket-oath and created an access_token, as described here.

This is something I've realised that was missing and I've done it before and got the same outcome, below snapchat:

 

[RuntimeException] 
Failed to execute git clone --mirror 'https://x-token-auth:***@bitbucket.org/company-name/my-repo.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-company-name-my-repo.git/'

Cloning into bare repository '/root/.composer/cache/vcs/https---bitbucket.org-company-name-my-repo.git'...
remote: Repository not found
fatal: repository 'https://x-token-auth:***@bitbucket.org/company-name/my-repo.git.git/' not found

 

The thing that got my attention was .git.git twice and closing with the message "remote: Repository not found".

My Satis config looks like this, there was ".git" in the end of each repo, but I've removed and nothing has changed:

"repositories": [
    { "type": "vcs", "url": "https://github.com/mycompany/privaterepo" },
    { "type": "vcs", "url": "http://svn.example.org/private/repo" },
    { "type": "vcs", "url": "https://github.com/mycompany/privaterepo2" }
  ],

 

Do you reckon this might be a bug on composer?

Maybe the old configuration is still in composer.lock? If the package was once fetched locally and committed, the package info is part of it IIRC.

I've refreshed the composer.lock and got the same. 
Also, I did a change on the pipeline. Cleaned the cache and forced it to use composer 2.0.8, everything has passed.

Cleaned again and used it 2.0.9, and got the same as above, the .git.git thing. It must be something on composer. I'll open an issue on composer github to confirm it. 

Thanks for you help. If I got a positive for a bug, or a config, I'll post here back.

Turns out there's an open issue on this topic, you can see here, the bug was introduced on composer version 2.0.9.

The pipeline set on my repo worked for a while due to cache (I guess), and once it expired, the pipeline started to fail.

The workaround for this issue is installing composer in the version 2.0.8

$ curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer --version=2.0.8

or update from the snapshot, as described here

$ curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
$ composer self-update --snapshot

or wait for 2.0.10 to be release. 

 

Thanks for you help @ktomk , it made me understand better what's happening behind the scene. 

Like ktomk likes this

Ah, you found a nice bug! If you have set-up the composer cache in the pipeline it can give these effects, they go away after seven days and then with the 2.0.9 version *zong*.

Btw. this is a reason why I can recommend to pin the composer version in a build pipeline (and also other builds). The only thing todo then is to check from time to time for a newer version and test it with a dedicated push (without cache naturally).

Was a pleasure to handle this together, was also nice for me and the .git.git was spot on. I have not noticed it in the original report. Troubleshooting.

Jordi from composer already confirmed the regression and the outlook for a fix looks good.

So lets keep the pipelines running!

And take a look if Private Packagist ist not an option. This supports composer as well and they have great additions compared to Satis for private setups.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Posted in Trello

Taco Tuesday: Your favorite Trello blog takeaway

Hello friends! From the community that brought you Welcome Wednesday, Throwback Thursday and Friday Fun, welcome to Taco Tuesday, a weekly discussion about all things Trello. The best part? One Tac...

67 views 4 3
Join discussion

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you