Cloning Bitbucket repo into a pipeline

[Felipe Luchi]

Hi,

I've seen soo many questions of this kind but any of them could solve my problem.

I have a repo A that pulls a repo B, both on Bitbucket. To make this work, I've created a SSH key for A and copy the its public key into Access keys on B.

Not sure if it matters but I've created my own packagist using satis (https://github.com/composer/satis).

From the composer logs, I can see this failing:

[401] https://bitbucket.org/foo/service/get/d70bca1785858d97e89aa553350ca9a19d797cd5.zip
Executing command (CWD): git config bitbucket.accesstoken
Failed to download my-vendor/service from dist: Could not authenticate against bitbucket.org
Now trying to download from source
- Syncing my-vendor/service (1.0.1) into cache
Cloning to cache at '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git clone --mirror 'https://bitbucket.org/foo/service.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git config bitbucket.accesstoken
No bitbucket authentication configured. Falling back to ssh.
Executing command (CWD): git clone --mirror 'git@bitbucket.org:foo/service.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'
Executing command (CWD): git --version
Failed: [RuntimeException] Failed to execute git clone --mirror 'git@bitbucket.org:foo/service.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git/'

Cloning into bare repository '/root/.composer/cache/vcs/https---bitbucket.org-foo-service.git'...
Resource not found
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

I'm really out of options, can't see any source that helps me somehow.

thanks

[ktomk · Accepted solution]

This looks like as if Satis or more likely Composer does not yet have these credentials and can't clone into its cache.

If this setup is within a pipeline that would puzzle me a bit honestly. Either you have a Satis setup and then Composer only needs to use it (that should then be without SSH as Satis uses HTTP/S) hence you should not see a git error.

However Satis here provides a source package then most likely and it does need the access then but default git SSH driver doesn't take it.

Are you able to clone directly with Git in the Pipeine already? At least for a test I would do it in front to more easily verify the SSH setup is correct. Satis/Composer may still fail, but you know the SSH configuration is good so far.

Next thing would be to check with Satis/Composer then and how to provide the SSH credentials so that Composer in the end can clone.

Some more Notes:

• Composer has some automatism when seeing a Bibucket URL. It may kick in here so if this is news to you, verify with verbose output that Composer is doing the things you expect or not.
• Composer has it's own auth/credentials mechanism, you may also need to add credentials to composer first, too (this might be especially true for source packages).

[Felipe Luchi]

Hi @ktomk , thanks for your reply.

About the question you did, the answer is yes. I can clone these private repos directly. 

I'm aware of SSH credentials and I had to that to clone from Satis on the server I work on. Setting these credentials on my pipeline was one of my attempts but it didn't work too.

Based on what you said, I can say that there are others layers that are not allowing pipelines to clone from Satis. The only one I can think of in this moment is the firewall. The server I set the Satis is protected via firewall, so even with these credentials set, the firewall doesn't allow the request to go through. 

Please, let me know your thoughts about that.

Thanks, Felipe 

[ktomk]

Hi Felipe, my question was if you can clone them in the pipeline. This should verify if transport is good or not before any Satis / Composer interaction.

The List of Public IPs from Bitbucket Pipelines, see here: https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/

[Felipe Luchi]

Hi @ktomk

I'm sorry I wasn't clear. I said yes when I try to clone them in the pipeline, I'm able to do it. 

I'll have a look at this link you sent and give you a feedback.

thanks

Felipe

[ktomk]

If cloning in the pipeline already works, the remote git repository is accessible via SSH.

The link then will not be that useful, as I put it in in case a firewall rule would have prevented SSH access to your remote server from the pipeline.

Now as SSH access works, and from the error message it is within composer install (right?), next thing to check might be composer.

The docs have a hint to SSH and private packages and authentication for private packages here:

• Handling private packages - Security 
• Authentication for privately hosted packages and repositories 

[Felipe Luchi]

From the second link, I've set bitbucket-oath and created an access_token, as described here.

This is something I've realised that was missing and I've done it before and got the same outcome, below snapchat:

[RuntimeException] 
Failed to execute git clone --mirror 'https://x-token-auth:***@bitbucket.org/company-name/my-repo.git.git' '/root/.composer/cache/vcs/https---bitbucket.org-company-name-my-repo.git/' 

Cloning into bare repository '/root/.composer/cache/vcs/https---bitbucket.org-company-name-my-repo.git'... 
remote: Repository not found 
fatal: repository 'https://x-token-auth:***@bitbucket.org/company-name/my-repo.git.git/' not found 

The thing that got my attention was .git.git twice and closing with the message "remote: Repository not found".

My Satis config looks like this, there was ".git" in the end of each repo, but I've removed and nothing has changed:

"repositories": [
    { "type": "vcs", "url": "https://github.com/mycompany/privaterepo" },
    { "type": "vcs", "url": "http://svn.example.org/private/repo" },
    { "type": "vcs", "url": "https://github.com/mycompany/privaterepo2" }
  ],

Do you reckon this might be a bug on composer?

[ktomk]

Maybe the old configuration is still in composer.lock? If the package was once fetched locally and committed, the package info is part of it IIRC.

[Felipe Luchi]

I've refreshed the composer.lock and got the same. 
Also, I did a change on the pipeline. Cleaned the cache and forced it to use composer 2.0.8, everything has passed.

Cleaned again and used it 2.0.9, and got the same as above, the .git.git thing. It must be something on composer. I'll open an issue on composer github to confirm it. 

Thanks for you help. If I got a positive for a bug, or a config, I'll post here back.

[Felipe Luchi]

Turns out there's an open issue on this topic, you can see here, the bug was introduced on composer version 2.0.9.

The pipeline set on my repo worked for a while due to cache (I guess), and once it expired, the pipeline started to fail.

The workaround for this issue is installing composer in the version 2.0.8

$ curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer --version=2.0.8

or update from the snapshot, as described here

$ curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
$ composer self-update --snapshot

or wait for 2.0.10 to be release. 

Thanks for you help @ktomk , it made me understand better what's happening behind the scene. 

[ktomk]

Ah, you found a nice bug! If you have set-up the composer cache in the pipeline it can give these effects, they go away after seven days and then with the 2.0.9 version *zong*.

Btw. this is a reason why I can recommend to pin the composer version in a build pipeline (and also other builds). The only thing todo then is to check from time to time for a newer version and test it with a dedicated push (without cache naturally).

Was a pleasure to handle this together, was also nice for me and the .git.git was spot on. I have not noticed it in the original report. Troubleshooting.

Jordi from composer already confirmed the regression and the outlook for a fix looks good.

So lets keep the pipelines running!

And take a look if Private Packagist ist not an option. This supports composer as well and they have great additions compared to Satis for private setups.