Pushing back to your repository

[#1]

[Feb 2019 update: All repos are now switched to using the http origin by default]

We’ve started rolling out an improved experience for committing back to your repository from within Bitbucket Pipelines - pushing back over http with zero git configuration.

Previously, you were able to commit back to you repository using SSH, which required manual configuration and management of SSH keys, often a time consuming and error prone process. Now you can commit back with zero configuration or management of keys.

This feature will be enabled by default for all new accounts, as well as those that haven’t pushed back using the SSH origin in any owned repositories within the past 6 weeks.

Pushing back over http from Bitbucket Pipelines is the preferred option, and will enable you to adopt new features of Bitbucket in the future.

Examples

Committing a changes to a branch

https://bitbucket.org/bitbucketpipelines/commit-a-new-file

pipelines:
  default:
    - step:
        script:
        - echo "Made a change in build ${BITBUCKET_BUILD_NUMBER}" >> changes.txt
        - git add changes.txt
        - git commit -m "[skip ci] Updating changes.txt with latest build number."
        - git push

Pushing a new tag 

https://bitbucket.org/bitbucketpipelines/pushing-a-tag

pipelines:
  default:
    - step:
        script:
        - echo "Made a change in build ${BITBUCKET_BUILD_NUMBER}" >> changes.txt
        - git add changes.txt
        - git commit -m "Updating changes.txt with latest build number."
        - git tag -am "Tagging for release ${BITBUCKET_BUILD_NUMBER}" release-${BITBUCKET_BUILD_NUMBER}
        - git push origin release-${BITBUCKET_BUILD_NUMBER}

How does it work?

Previously, the git ‘origin’ was configured to use SSH for authenticating with Bitbucket, with the git origin set to ‘git@bitbucket.org/<<your account>>/<<your repo>>.git’. To commit back, you had to configure a git username, email, SSH public/private key pair, and have either pipelines pass the keys to git, or configure it yourself, manually. 

By default, all repositories that haven’t recently used SSH to push will now have the git origin set to ‘http://bitbucket.org<<your account>>/<<your repo>>’, and will take advantage of fully managed authentication. We send git requests to Bitbucket via a proxy which runs as part of your pipeline. We inject an auth token using a Git proxy that runs along side your main build container, into all Bitbucket git requests which are send via https to Bitbucket, allowing you to seamlessly and securely perform git operations.

Limitations

We don’t want anyone currently using SSH to be impacted by this change so we’ve added all accounts that have recently used SSH to an exclusion list. In this case, all repositories under your account will continue to use the SSH origin until February 1st 2019, when we plan to switch ‘origin’ from SSH to the new http mechanism.

Pushing back using the http origin to repositories where branch permissions are enabled is not yet supported. In this scenario you will need to manually configure git to use an alternate authentication method, providing the necessary credentials that allow you push back to the repository. For these situations we recommend pushing over http using OAuth based authentication. See our docs for details on how to set up alternative authentication methods.

I've been using SSH and am excluded from the feature - how do I switch?

I want to use the new feature

You will need to manually configure the git origin until you've been removed from the exclusion list. To do this, execute the following command from within your bitbucket-pipelines.yml file to set origin to use http:

https://bitbucket.org/bitbucketpipelines/git-auth-use-http

git remote set-url origin ${BITBUCKET_GIT_HTTP_ORIGIN}

Once you've been removed from the list, you can revert this change.

I want to keep using SSH

To continue using SSH, you need to manually set the origin to SSH by executing the following command from within your bitbucket-pipelines.yml file:

https://bitbucket.org/bitbucketpipelines/git-auth-ssh-configured-in-ui/

git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}

Take me off the exclusion list

If you are no longer relying on the SSH origin and wish to switch to http, you can request to get off the exclusion list by submitting a request. Once this is done, the http origin will become the default for your account, and you can remove any manual configuration.

Using ‘origin’ variables

We’ve introduced 2 new variables that can be referenced from within pipelines, to contain the http and SSH origin urls. We recommend using these where the origin needs to be passed through to scripts or other tools that are invoked in your pipeline, or if you want to switch between http and SSH:

${BITBUCKET_GIT_HTTP_ORIGIN}

${BITBUCKET_GIT_SSH_ORIGIN}

Configuring an alternate Git client

If you are not using the git CLI provided by Pipelines, we recommend configuring origin using the ${BITBUCKET_GIT_HTTP_ORIGIN}  variable. If using the http origin, you will also need to configure your git client to use a proxy. The proxy url is: http://localhost:29418.

[Malte Hübner]

@Linette Thank you for this great improvement! Currently we are using SSH to push back. And we are checking out other branches in our build process (i.e. development to commit a snapshot version increment). But the new feature seems not to support such operations (the checkout cancels the build process). Do I have to configure any access controls?

[Graham Gatus]

@Malte Hübner the feature is still in the process of being rolled out, and we expect all accounts (except those currently pushing with SSH, which will be notified with an in app notification once rollout has complete) to have it within the next 24 - 48 hours.

If you wish to checkout another branch and commit back, the following should work:

pipelines:
  default:
   - step:
        script:
          - git fetch origin develop:develop
          - git checkout develop
          - echo "${BITBUCKET_BUILD_NUMBER}" >> changes.txt
          - git add changes.txt
          - git commit -am "[skip ci] Made a change in ${BITBUCKET_BUILD_NUMBER}"
          - git push

If you still have issues, feel free to raise a support ticket at https://support.atlassian.com/contact/#/ and we can investigate further.

[Malte Hübner]

@Graham Gatus Thanks for looking into it. I'll give it another try on Thursday when rollout should have been done. Regarding SSH and excluded repos I added:

git remote set-url origin ${BITBUCKET_GIT_HTTP_ORIGIN}

So this part should not be an issue.   

[Jono]

Hi there, thanks for the awesome feature!

@Linette or @Graham Gatus - can you elaborate on the branch permissions limitation? Does having branch permissions enabled on the repo mean that it won't work at all or branch permissions that limit write access to select users?

[Peter Morlion]

Hi, I saw the message when I was going to one of our company repositories. It would be useful if the message would mention which repository did the push using SSH. Currently, I have no way of knowing which repository I should be looking at. This isn't a problem if you only have a few repositories, but if you have many, and don't know all of them (history, legacy, reasons), it's a bit harder. Would it be possible/easy for you to list which repositories users should look at?

[Deleted user]

 Displaying which repos will be affected of this change will save a lot of time rather than saying
"A repo in your account pushes back using SSH origin. Your configuration will stop working from 1st February 2019, unless you make some small changes. To help you switch here are the details of what you need to do next."

Thanks in advance :)

[Cedric van Putten]

We have the same issue as @Jono, the branch permissions.

Right now we use SSH keys attached to a user, which has explicit write access to protected branches; `develop` and `master`. Other developers may create PRs from any (feature) branch, but may not write to these two protected branches. I can't find any user or entity to give this explicit access. We really need this for us to switch over to HTTP pushes.

I would love to see either a checkbox to allow CI to write to these branches. Or a user/entity to select in the `write access` box.

[NPxComplete]

Thanks BB Team, very excited to have this feature finally. Gotta know though, will I be able to permission my pipeline to push to some other repo I own?

An example use case,

For some of my OSS projects I try to push coverage reports to my <user>.bitbucket.org repo after every minor version bump. 

How would you recommend we implement this now, and how can we hope to do this in the future?

[ryanbannon]

Hi,

I'm in the same situation as @Jono and @[deleted]. We have a special CI/CD user that is used in our Bitbucket Pipelines to push tags to our develop/release/master branches. @[deleted] has an interesting suggestion; I kind of like the idea of having a checkbox that allows pushes over HTTP from within BP. Although maybe there are security concerns?

[Graham Gatus]

@Peter Morlion  / @[deleted], please contact support and we can provide information about the repositories you own that were pushing via SSH leading up to the 9th December.

[Graham Gatus]

@Jono / @[deleted] / @ryanbannon: Branch permissions can be applied and will work if the repository owner has permissions to write to the branch being pushed to, however this doesn’t work if the repository is created under a Team account. The complication here is that pipelines executes using the account UUID of the repository owner (which for a repo under a Team account, is the team UUID), and Bitbucket has no mechanism to configure branch permissions for the Team itself, only for groups of users created within the Team.

A feature is being planned to allow first class support for Bot accounts, which will allow pipelines to execute as a dedicated bot user whilst simultaneously allowing branch permissions to be applied. Stay tuned for updates.

In the meantime, you can continue using SSH, just ensure you set your origin before Feb 1st, after which origin will be switched over to HTTP.

[Marcy Nugent]

I switched our repos to using an OAuth consumer to prevent breaking builds, and now the OAuth consumer can't push back to repo with branch permissions - is there any way to push back to branch protected repos now that SSH is going away?

We had to use the App Password method since we had pulled out the SSH configs to comply with this change - it's really not ideal to expose a user's account to Pipelines like that, but I'm not seeing a better option for now.

[Graham Gatus]

@NPxComplete, pushing back to other repositories under your account is not  supported by this feature. You would need to manually configure one of the alternate authentication methods  to allowing pushing to another repository. Discussing with the team, this may be something we could consider when we allow first class support for Bot accounts in the near future.

I've created a site master ticket to track this request - feel free to comment / vote up! - https://bitbucket.org/site/master/issues/17799/allow-pushing-to-other-repositories-under

[ryanbannon]

@Graham Gatus thanks for the response. Yeah, we're under a team account, so that's exactly our scenario. Good to know something is being worked on. THANKS! :)

[Vincent Truong]

I was just looking to do something like this today.

Does this zero-config work using Mercurial? Hope we are not forgotten :)

(Assuming it does work - this is all I need right?)

script:
  - hg tag "my/new/tag/$(date +%Y-%m-%d)"
  - hg push

[Andrew Stec]

What are the key offending lines of code that Bitbucket is looking for in the pipeline build file?

The problem is that: 

- I do not know the key repos that are causing this.

- I have looked through each repo and I do not see Git pushing. 

- I am authenticating using eval (with SSH keys and AWS authentication), and I wonder if this is triggering the message even though I am not pushing to Git.

[ryanbannon]

@Andrew Stec I imagine that they're not scanning code but have a proxy setup that's checking if any POSTs have been made back to Bitbucket IPs from the associated BP containers.

[Graham Gatus]

@Andrew Stec the warning message is shown for all accounts where a pipeline executed that committed back to a repository over SSH, during a 6 week period leading up to the feature being enabled.

[Graham Gatus]

@Marcy Nugent SSH will continue to work, however the SSH origin will no longer be the default we set after 1st February. To continue using SSH after this date, you can manually set the origin to the SSH origin in your bitbucket-pipeline.yml file by executing:

git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}

Please raise a support ticket if you have issues using any of the other available authentication methods and we can investigate.

[Beatriz Capelli]

Hello, I was waiting for the feature and just now I'm giving it the first try.
But I have a dummy question now that it is finally released - won't Bitbucket fall into an infinite loop if pushing back from Pipelines?
It will run again because of the commit being pushed internally.
Below is what happened here (nevermind the failures - diff issue)
I will now start to look for a workaround.

[ryanbannon]

@Beatriz Capelli yes, it would, but only if you program the pipeline to do that. :)

I suppose it all depends on how you want your overall pipeline to behave. One thing that I do is push back to the repository only to TAG a commit. (In fact, I would argue that it is bad practice to push content changes back to the repo from within a pipeline. I'd stick with the rule that content is only pushed into a repo via a user.) I use tags to mark commits that have run through the pipeline successfully. I then have a separate pipeline that is triggered when a commit is tagged. That pipeline does a publish to a registry like Nexus.

It should be noted, however, that I'm abandoning this practice in favor of a single multi-step pipeline (for various reasons).

[Beatriz Capelli]

@ryanbannonI see. I wanted to bump my "version.sbt" file via Pipelines, pushing back only this change, that's why I was waiting for it, but I won't be able to do it then.

I will look for an alternative, thank you for the reply!

[ryanbannon]

@Beatriz Capelli My pleasure :)

Your comment implies that you're using pipelines to manage versioning, yes? Question: are you also publishing artifacts to some kind of registry after you do a build?

[Beatriz Capelli]

@ryanbannonThat is the idea, yes, but our publish/release workflow needs to push to repository 2x before done.

I was waiting for the issue fix to be able to automate it, but apparently we will need to run the release from our local machines before pushing, and to use Pipelines only for merging to a release branch or something.

[ryanbannon]

@Beatriz Capelli I'd be curious to know about your pipeline, if you're interested in sharing.

@Linette in fact, I'd be interested to hear about how other people are using BP. What's the best forum for such a discussion?