Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Problem getting BitBucket Pipelines to commit back version change to its own GIT repo

Bryan Copeland March 27, 2018

I'm trying to enable BitBucket Pipelines for a Java-based project to auto-increment the build number following Semantic Versioning. The version incrementing is in place and working well via Maven Versions plugin, the only thing I need to get working is BitBucket Pipelines / Docker image to be able to commit back to the repository...

Here's the relevant section from my BitBucket Pipeline YAML config:

 



image: maven:3.5.2-jdk-8
pipelines:
  default:
    - step:
        caches:
          - maven
        script:
          - mvn -B -U clean install
    - step:

          #increment all code version numbers
          - snapshotversion=`grep 'build.version' build.properties | awk -F '=' '{print $2}'`
          - releaseversion=`echo "$snapshotversion" | cut -f1 -d"-"`
          - mvn versions:set -DnewVersion=$snapshotversion
          - mvn versions:commit
          - mvn -B -U clean install
                              
          # Commit any changes back to this "release/x.y.z"... definitely good to here, the rest meh :(
          - git add .
          - git config user.name "$GIT_USERNAME"
          - git config user.email "$GIT_EMAIL"
          - ssh-copy-id -i ssh-rsa $BITBUCKET_SSH_PUBLIC_KEY git@bitbucket.org:${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git
          - ssh -i ~/.ssh/config git@bitbucket.org:${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}.git
          - commitmsg="[skip ci] Release Candidate for merging to master branch and PROD deployment"
          - if [[ -n $(git status -s) ]] ; then filelist=`git status -s` ; git commit -a -m "$commitmsg" -m "$filelist" ; git push origin release/$releaseversion:release/$releaseversion ; else echo "No changes detected"; fi

 

 

 The error it gives is:

/usr/bin/ssh-copy-id: ERROR: failed to open ID file 'ssh-rsa.pub': No such file


So I think its fairly close, the only part I can't figure out is how to add reference to my SSH key (either by file or injecting the whole Public Key as an environment variable for instance), what I'm trying to do is clearly wrong since it errors out. Does anyone have a clear working example of committing ANYTHING back to a BitBucket GIT repo via BitBucket Pipelines?

I've followed the steps suggested here by creating an SSH Key within my BitBucket Cloud settings for the repo, but with no luck:


https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html

Should the SSH Key really be "automagically available" to its own repo's Pipeline, like some reported, or like others report do you need a very kludgy workaround:\

https://community.atlassian.com/t5/Bitbucket-questions/Trouble-with-SSH-and-Bitbucket-Pipelines/qaq-p/604330

https://bitbucket.org/site/master/issues/13213/push-back-to-remote-from-pipelines

Appreciate these threads as it helps to get suggestions from the community and/or Atlassian. Hoping someone can spot my stupid mistake and that last little push to CD wonderland hah... but seriously, really appreciate any support on this!

6 answers

2 accepted

2 votes
Answer accepted
Philip Hodder
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 3, 2018

Hi Bryan,

Unfortunately a kludgy workaround is indeed currently needed.

Here's the steps to get repo push-back in Pipelines working.

  1. Set up a Pipelines SSH key as specified in Step 1 of: https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html?_ga=2.166794103.859441905.1522715061-825437565.1515570924
  2. Add the public key of the SSH key you created to a Bitbucket Account's SSH keys. (Account Settings -> Security -> SSH keys). I'm unsure if there's a better way to set this up so it's not tied to an account. Otherwise you can create a dummy account if you don't want it linked to a specific team members account. Related docs here: https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html
  3. Repo push back should now be possible. Try the following yaml.
image: ubuntu:16.04
pipelines:
default:
- step:
script:
- apt-get update -y
-
apt-get install -y git
-
echo "Hello" >> file.txt
-
git add .
-
git commit -m 'I said hello [SKIP CI]' # [SKIP CI] prevents builds being triggered off this commit.
- git push origin master

Thanks,

Phil

Bryan Copeland April 18, 2018

Thanks very much for the response @Philip Hodder I tested this out in our most recent release branch and unfortunately I'm still getting this error within BitBucket Pipelines:

Host key verification failed.fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.

Is there somewhere in specific that I need to "register" my SSH Key for once within BitBucket Pipelines for it to be aware how to commit back to its own repo?

Also, I generated the SSH Key at the repository level as the documentation seemed to show, then tried to add it to my personal account but it says:
Someone has already registered this as a deploy key.

Again I set this up under the repository itself --> Settings page is that incorrect? Do I need to use one of my own personal SSH Keys instead or anything else I might be missing?

Like Cory Root likes this
Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 7, 2018

Hi @Bryan Copeland,

I'm the new writer for Bitbucket Pipelines and I hope I can help here!

Firstly know that we are looking into this, both to improve the feature, and the documentation. We appreciate this can be a pain point. :(

So, on to an explanation:

There are 3 places you can enter an SSH key into the system, and how you decide to set that up depends on what you need to do:


1) User SSH key
( Click your face in the bottom left > Bitbucket settings > (under the Security heading) SSH keys)

A key you put in here will identify who you are, and give read and write access to all your available repositories.


2) Repository Access keys
(In your repository click Settings and under the General heading select Access keys)

Any key you put in here will have read only access to the repository (this also used to be known as a 'deploy key'). As any key entered here is only allowed to read, you can't use the same key here as you used for 1 (which has r/w). Some people never enter a key here.

 

3) Pipelines SSH key
(In your repository click Settings and under the Pipelines heading select SSH keys).

This is the key that Pipelines uses to interact with your repository and identify itself.

If you use the same key you used in 1), you'll have read and write access, BUT all the logs will say that you, Bryan, have been doing all the things that Pipelines is actually doing. Also, this key can't use a passphrase, which can be another issue if you are thinking of using your own user SSH key.

If you use the same key you've used in 2) then you'll have read only access and won't be able to push back.

Sooo, what some people do, if they want to have write access to the repo, is to make a bot Bitbucket account with access to the relevant repo. They enter in the same key (that doesn't use a passphrase) in 1) and 3) and then they have r/w access and all the logs show the bot username.

I hope that makes things a bit clearer (and if that explanation helps, I'll add it to the docs).

1 vote
Answer accepted
Bryan Copeland June 14, 2018

Hi @Linette thanks very much for the response, I believe I've tried all three of these approaches with no luck, but I'm going to give it another crack with the info in #2 that the "Repository Access Key", which was my latest attempt, is read-only. That was the approach I tried most recently and could explain the recent failures... I'll also try creating a new "Bot/System" user account. Although unfortunately it incurs a slightly higher monthly cost which is somewhat frustrating, the benefits in terms of man hours spent bumping the version each release will likely more than make up for the new costs.

Thanks, I'll report back here as soon as I know the outcome...

Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 27, 2018

Hi @Bryan Copeland, did you get a chance to try this out in the end?

Bryan Copeland June 29, 2018

Hi @Linette sorry I must be a bit daft, but I'm not understanding where to put the new Bot user's SSH key in order for Pipelines to read it, is that just within the whole project's SSH keys listing, somewhere within Pipelines, elsewhere?

Also, if I make a change as myself that creates the situation (trigger) where we want to push back to our repo (in our case just following GITFLOW so at the moment I create a "release/x.y.z" branch), how will the Bot user's account be the one attributed to bumping the version and pushing that change back to the repo?

Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 9, 2018

Hi @Bryan Copeland, you're not being daft, it's a confusing area!!

Probably the most straightforward order of operations would be:

  1. Create a bot user account in BB

  2. As a repo admin, give the bot user account write permission to the repository in Repo settings > general > user and group access
  3. Go into Repo settings > Pipelines > SSH keys

  4. Generate a key pair and then copy the public key (this is the key pipelines will use to identify itself)

  5. Switch to the bot account and navigate to Bitbucket settings >Security >  SSH keys

  6. Paste public key (we are giving the bot the same ID that we are using for pipelines)


Your bot user will be credited with making the changes, as pipelines will be using the bot user's SSH key to clone your repository and perform the commands and writes. Essentially you make a trigger happen, and pipelines takes it from there.


Hope that makes things a little clearer?

Like Bryan Copeland likes this
Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 24, 2018

@Bryan Copelanddoes the explanation above help?

Bryan Copeland August 12, 2018

@Linettethanks very much for the ongoing help with this. Apologies for the delay, but I've been waiting large 2-week gaps to retry little tweaks to the process (which is how often we cut a "release/x.y.z" release candidate branch upon which I try to bump the versions).

I thought I could just modify my existing setup and achieve what you described above, but each attempt was still falling short. For our next release, I've gone back to the drawing board and created a brand new bot account doing the exact steps you've listed.  When our next "release/x.y.z" branch gets created near the end of the week, I'll confirm whether it worked or not.

If it didn't work this time, I might be hopeless. I wonder if there's any possibility of getting a WebEx type session with yourself or someone on BitBucket support to review our setup in the worst-case?

Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 13, 2018

oooh! I'll keep my fingers crossed for you!!

I'm sure we can figure out something if that doesn't work for you, but here's hoping it all works like a charm!

Bryan Copeland August 15, 2018

Thanks so much Linette, this finnnnally works!!!!

Going to be writing a blog post about Automated Versioning with Maven and Pipelines at some point soon with a link to this and your awesome advice (if you don't mind).

Thanks again!

Linette
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 15, 2018

What great news @[deleted]!!! Feel free to link and I'll also update our documentation to try and make this all a bit clearer! :)

Deleted user September 19, 2019

@LinetteI have tried this approach but using my own account rather than a bot account (because I don't know how to create one of those), and the pipeline is still unable to commit to the repo.

Having read about it elsewhere I also tried setting the public key as an access key on the Repo  Repo > Settings >  Access keys but sadly that did not work either because this only grant read-only access

Kenny Deckers October 17, 2019

I was experiencing the same problem and found out that using hirak/prestissimo for parallel cloning was causing the problem. Removing it makes the approach outlines above work.

Erwin Vrolijk January 22, 2020

@Linette There is one thing missing from your answer, which we needed to get this to work. In the default image used in a pipeline the repository is checked out via http. So any change to ssh keys will not affect the push.

On top of your solution I've added the equivalent of the following to our pipeline:

- git add .
- git commit -m "Automatic update from pipeline"
- git remote add ssh $BITBUCKET_GIT_SSH_ORIGIN
- git push ssh HEAD

I add a new remote to the repo, and call this remote 'ssh'. The url of this remote is the git string to this repo.

Erwin Vrolijk January 31, 2020
christianmartos February 13, 2020

Hi @Linette ,

could you please explain how to create a bot account in BB.

As you are describe it above it sounds super easy for someone how knows what to do. But for me as a "not DevOp" it is hard to find out what to do.

Would be great if you or anyone else could help me with this problem.

Thank you.

0 votes
Kerry Johnson May 7, 2021

I use a CLI tool called Lerna, and I then have Lerna run another CLI tool called gitpkg.  When gitpkg would attempt to push tags to BitBucket, it would error out with the following:

fatal: could not read Username for 'https://bitbucket.org': No such device or address

 
I tried many solutions listed here and elsewhere with setting up keys, app passwords, etc. but could not work around this issue.  So I looked into the build process and noticed that BitBucket Pipelines sets its own git credentials ahead of time:

git config user.name bitbucket-pipelines
git config user.email commits-noreply@bitbucket.org
git config push.default current
git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/


I noticed that git commands issued directly (as in not issued through another CLI tool within) ran perfectly.  So I had a suspicion that there might be a permission issue or some walled garden at play when Gitpkg tries to push its tags.

I tested my theory by re-issuing the git config lines, but expanding the config scope globally in hopes Gitpkg would see it

branches:
master:
- step:
script:
- git config --global user.name bitbucket-pipelines
- git config --global user.email commits-noreply@bitbucket.org
- git config --global push.default current
- git config --global http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy http://localhost:29418/
- nvm install v12
- npm install
- lerna publish --yes


It worked!  I know my issue might not be exactly the same as yours but maybe someone out here with the same as issue as me will come across this one day.

 

--

 

Edit:  The above works so long as your branch isn't protected.  In order to have this work with protected branches, you will need to look at "SSH Key pair managed by Bitbucket Pipelines": https://support.atlassian.com/bitbucket-cloud/docs/push-back-to-your-repository#SSH-Key-pair-managed-by-Bitbucket-Pipelines

Then set it like:

git config --global user.name bitbucket-pipelines
git config --global user.email commits-noreply@bitbucket.org
git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}

as in

branches:
master:
- step:
script:
- git config --global user.name bitbucket-pipelines
- git config --global user.email commits-noreply@bitbucket.org
- git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}
- nvm install v12
- npm install
- lerna publish --yes 

 and it should push without getting the "Permission denied to update branch" error.

0 votes
Deleted user September 19, 2019

I want to know how to set up a bot account, when I try to add a user I only get the option to do so with an email address.

Kenny Deckers October 17, 2019

I have also looked into this and currently this seems the only way to create a bot account.

Deleted user October 17, 2019

We ended up creating an AD distribution group and registering that with BitBucket; the issue for me on this is that it uses one of our licenses when there are no actual users attached.

0 votes
Deleted user August 14, 2018

Hi all, I was able to solve this (working around my shh key using bitbucket pipelines) by creating an app_password and using that instead of my personal password.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events