Changes to Bitbucket API Requires Latest Version of Go

[#1]

As part of our efforts to continuously enhance Bitbucket Cloud’s security, we’re changing response codes for some requests to private repositories. Users who do not have access to a private repository will receive 404 (not found) response codes instead of 403 (forbidden) response codes to mask the existence of a private repository at a given URL path.

Rolling out these changes will break previous versions of Go due to the fact that the go command relies on a 403 response to fetch repositories hosted on Bitbucket Cloud. This means that users who use older versions of Go with private repositories, for example CI/CD builds with Go dependencies, will run into 404 errors.

Changes were recently made to GoLang’s source code here to unblock Bitbucket’s rollout of the security patch. To ensure that you will not run into any issues, update GoLang to the latest version.

1. Ensure you are on a supported version of GoLang: 1.18, 1.17, or 1.16.

2. If you are using 1.17 or 1.16, ensure you are on the latest patch (1.17.7 or 1.16.14)

Update: We are rolling this out as of June 1st 2022, following the email notification we sent out to affected users on May 9th 2022.

[Sam Wood]

Glad you're improving the security, but wow this caught us off guard. 

[sneko]

To be honest, I don't understand why you didn't make the effort of parsing your users' repositories to see if they contain a "go.mod" file. And then, send to their admin an email to inform the Golang programs will need specific adjustments. How am I supposed to see this community article easily?

I did the investigation from scratch, trying to "git clone" from SSH, then from HTTPS, then without "GOPRIVATE"... then with public Golang Bitbucket repositories... then suspecting Golang and found this article when searching. Ok, no more than 1 hour hopefully but... I feel the pain for all other people who will face this. (moreover, we had a big doubt because your Cloud service was really slow (minutes to load a page, and we got errors too... so easy to suspect this instead of continuing investigating)

Hope you will do it better next time!

[Akhil Durge]

Was it announced anywhere before? Our bitbucket pipelines are failing & it's a blocker at the moment. People can't just jump to next GoLang version without testing.

Is there any quick alternative?

[yoavd]

Same here. We also spent ~1 hour troubleshooting ssh keys / git config / GOPRIVATE configurations before reaching out to BitBucket support who immediately directed us to this community article...

Moreover, our (>100) repositories use the circleci:golang set of images - these were NOT updated to the latest go version which includes the latest patch (e.g. 1.17.7 or 1.16.14).
As a result, all of our our pipelines now fail and we need to spend more time to update them use cimg/go images.

It is unfortunate that BitBucket did not announce these breaking changes in advance, and/or worked with CircleCI to update the circleci/golang images to use the latest minor version (that includes the patch).

[Ramesh Naik Banothu]

Our bitbucket pipelines are getting failed.

We are trying for this for 6 hours by debugging our Private keys and all.

It's not even intimated by bitbucket.

People can't just jump to the next GoLang version without testing.

Is there any alternatives?

[jlxm]

Very unfortunate. Could have handed better. We spent numerous hours trying to debug on pipeline and private and security keys, until we tried to send a support ticket.

They are suggesting we upgrade to GoLang: 1.18, 1.17.7 or 1.16.14

We are also not sure if everything will work right away.

This is very sad and frustrating specially if people are deploying.

[michael.renz]

Wow, this was handled horribly.

And is a big part of the reason we're migrating away from bitbucket at the moment.

Of course, this means that we need to support all of our developers by -without notice and right at the end of the quarter- switching all of our work over to making sure our build images work with the latest versions of golang, and put effort into otherwise dead projects to do so.

Thanks for the last-minute spike of work!

[Alvaro Molano]

@Luke Choe I just read a ticket response from your team. Upgrading go is not something we want to do in a hurry.

I tried adding .netrc adding (machine api.bitbucket.org and login credentials) to our pipelines without needing to update go and so far it looks like it will solve the problem for us. 

Best!

[Madura Dissanayake]

We cannot upgrade go version without testing and our prod pipelines are failed completely, please give us a solution to work with our existing versions of Go.

[Luke Choe]

IMPORTANT FOLLOW UP: 

Our communication strategy for this breaking change was definitely not ideal in terms of visibility. We've rolled back our security patch, and we are going to re-discuss how to better inform users and/or explore alternative solutions.

We apologize for the trouble this has caused.

Feel free to reach out here with any questions/concerns in the meantime.

Best,

Luke 

[Atlas Peng]

hello 👋 
just wondering if updating Go version to the current latest patch i.e. 1.17.8 or 1.16.5, will these also work?
current and previous versions can be found here https://go.dev/dl/

or will we still receive the 404 error if it's not using version 1.17.7 or 1.16.14?

thank you and cheers

[Luke Choe]

@Atlas Peng Yes, future versions that are at least 1.17.7 and 1.16.14 will also work. 

[Massimiliano Monaro]

If anyone is using AWS Codebuild (which is currently locked to 1.16.4) I opened a PR to update Go https://github.com/aws/aws-codebuild-docker-images/pull/519 

[JH]

Wow we wasted a lot of time on figuring out what happened here. Couldn't you guys have scanned pipeline yaml files for go images and notified those folks?

[Andrew Jones]

Does this break access for anyone using >= go 1.15?  We still have no access after the rollback?

Edit: Just read the update, it's been rolled out which explains why our pipelines broke.  No email on the 9th of May as suggested, other commenters suggest they also did not receive this either.  Obscure blog post to announce the breaking change, rollback for 1 week? then forward again.  

[Michiel Nugter]

Again this feature has been rolled out. This completely breaks any integration with AWS Lambda as it's still stuck on an unsupported Go version.

Please revert this as soon as possible again and hold on releasing as this makes using bitbucket repositories on AWS Lambda / CodeBuild impossible.

[Manjunath S V]

Did you guys rolled back the  changes, Looks like its working now.

Or What I did is 

export GOPRIVATE="bitbucket.org/org-name"

[Michiel Nugter]

We have a temporary work-a-round implemented in our code-build script.

We added the following in the install phase commands.

// TEMPORARY WORKAROUND to install latest minor fix of go
// remove after this merged https://github.com/aws/aws-codebuild-docker-images/pull/519
'wget -q https://go.dev/dl/go1.16.15.linux-amd64.tar.gz',
'tar -C /usr/local -xzf go1.16.15.linux-amd64.tar.gz',
'export PATH="/usr/local/go/bin:$PATH"',
"go version",

I recommend to keep using version 1.16 as you will otherwise run into problems running it in the lambda function.

[Massimiliano Monaro]

Some of our pipelines are broken again.

We need to push AWS to this PR https://github.com/aws/aws-codebuild-docker-images/pull/519 so we can finally use standard (updated) codebuild image.

[Feng Xiao]

where is the announcement?

[david]

I've had all of our Org Admins search their inboxes and there was no announcement sent to any of them regarding this issue.  How on earth were we expected to appropriately respond ?  This took our deployments down for over a business day while we quickly hacked together a fix.  Please provide a better story for how communications will be handled in the future.

[Andrew Jones]

If Bitbuckets definition of "affected users" is noone then their comms were excellent.  We've searched for the email also but can't find it.  Pretty silent thread on Bitbuckets side here given the breaking change they introduced.  Where is the announcement for the second roll out?  They gave us an excuse to start our migration to Gitlab, one service done and tbh Gitlab is amazing so far.  It's significantly faster, has CI/CD built in, each repo has a docker registry.  Just wish we'd done it earlier.

[Akhil Durge]

Hi @Luke Choe ,

We have upgraded GoLang version to 1.17.11 in production. If you are planning to revert your changes, please make sure nothing breaks on this version.

Thanks,

Akhil

[Massimiliano Monaro]

FYI AWS merged the upgrade for GO 1.15.15 and 1.16.15 https://github.com/aws/aws-codebuild-docker-images/issues/518 leaving my PR open for the next 1.17 and 1.18 https://github.com/aws/aws-codebuild-docker-images/pull/519/ (hoping they don't take more months of waiting)

[N1ghteyes]

This change seems to have broken some composer builds on our side as well - Specifically using Satis composer server.

The 404 status change means that we now get exceptions rather than login prompts or fall-back to SSH - this is baked into the core composer transport package, so not something we can fix quickly.

Completely unnecessary breaking change IMHO.

EDIT:: We've begun migrating to Gitlab. Breaking changes of this nature are entirely unacceptable.