Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,463,419
Community Members
 
Community Events
176
Community Groups

Changes to Bitbucket API Requires Latest Version of Go

As part of our efforts to continuously enhance Bitbucket Cloud’s security, we’re changing response codes for some requests to private repositories. Users who do not have access to a private repository will receive 404 (not found) response codes instead of 403 (forbidden) response codes to mask the existence of a private repository at a given URL path.

Rolling out these changes will break previous versions of Go due to the fact that the go command relies on a 403 response to fetch repositories hosted on Bitbucket Cloud. This means that users who use older versions of Go with private repositories, for example CI/CD builds with Go dependencies, will run into 404 errors.

Changes were recently made to GoLang’s source code here to unblock Bitbucket’s rollout of the security patch. To ensure that you will not run into any issues, update GoLang to the latest version.

  1. Ensure you are on a supported version of GoLang: 1.18, 1.17, or 1.16.

  2. If you are using 1.17 or 1.16, ensure you are on the latest patch (1.17.7 or 1.16.14)

 

Update: We are rolling this out as of June 1st 2022, following the email notification we sent out to affected users on May 9th 2022.

26 comments

Glad you're improving the security, but wow this caught us off guard. 

Like # people like this

To be honest, I don't understand why you didn't make the effort of parsing your users' repositories to see if they contain a "go.mod" file. And then, send to their admin an email to inform the Golang programs will need specific adjustments. How am I supposed to see this community article easily?

 

I did the investigation from scratch, trying to "git clone" from SSH, then from HTTPS, then without "GOPRIVATE"... then with public Golang Bitbucket repositories... then suspecting Golang and found this article when searching. Ok, no more than 1 hour hopefully but... I feel the pain for all other people who will face this. (moreover, we had a big doubt because your Cloud service was really slow (minutes to load a page, and we got errors too... so easy to suspect this instead of continuing investigating)

 

Hope you will do it better next time!

Like # people like this

Was it announced anywhere before? Our bitbucket pipelines are failing & it's a blocker at the moment. People can't just jump to next GoLang version without testing.

Is there any quick alternative?

Like # people like this

Same here. We also spent ~1 hour troubleshooting ssh keys / git config / GOPRIVATE configurations before reaching out to BitBucket support who immediately directed us to this community article...

Moreover, our (>100) repositories use the circleci:golang set of images - these were NOT updated to the latest go version which includes the latest patch (e.g. 1.17.7 or 1.16.14).
As a result, all of our our pipelines now fail and we need to spend more time to update them use cimg/go images.

It is unfortunate that BitBucket did not announce these breaking changes in advance, and/or worked with CircleCI to update the circleci/golang images to use the latest minor version (that includes the patch).

Like # people like this

Our bitbucket pipelines are getting failed.

We are trying for this for 6 hours by debugging our Private keys and all.

 

It's not even intimated by bitbucket.

People can't just jump to the next GoLang version without testing.

Is there any alternatives?

Like # people like this

Very unfortunate. Could have handed better. We spent numerous hours trying to debug on pipeline and private and security keys, until we tried to send a support ticket.

 

They are suggesting we upgrade to GoLang: 1.18, 1.17.7 or 1.16.14

 

We are also not sure if everything will work right away.

This is very sad and frustrating specially if people are deploying.

Like # people like this

Wow, this was handled horribly.

And is a big part of the reason we're migrating away from bitbucket at the moment.

Of course, this means that we need to support all of our developers by -without notice and right at the end of the quarter- switching all of our work over to making sure our build images work with the latest versions of golang, and put effort into otherwise dead projects to do so.

Thanks for the last-minute spike of work!

Like # people like this

@Luke Choe I just read a ticket response from your team. Upgrading go is not something we want to do in a hurry.

I tried adding .netrc adding (machine api.bitbucket.org and login credentials) to our pipelines without needing to update go and so far it looks like it will solve the problem for us. 

Best!

Like # people like this

We cannot upgrade go version without testing and our prod pipelines are failed completely, please give us a solution to work with our existing versions of Go.

Like michael.renz likes this
Luke Choe Atlassian Team Mar 24, 2022

IMPORTANT FOLLOW UP: 

 

Our communication strategy for this breaking change was definitely not ideal in terms of visibility. We've rolled back our security patch, and we are going to re-discuss how to better inform users and/or explore alternative solutions.

We apologize for the trouble this has caused.

Feel free to reach out here with any questions/concerns in the meantime.

 

Best,

Luke 

Like # people like this
Atlas Peng Atlassian Team Mar 27, 2022

hello 👋 
just wondering if updating Go version to the current latest patch i.e. 1.17.8 or 1.16.5, will these also work?
current and previous versions can be found here https://go.dev/dl/

or will we still receive the 404 error if it's not using version 1.17.7 or 1.16.14?

thank you and cheers

Luke Choe Atlassian Team Mar 28, 2022

@Atlas Peng Yes, future versions that are at least 1.17.7 and 1.16.14 will also work. 

If anyone is using AWS Codebuild (which is currently locked to 1.16.4) I opened a PR to update Go https://github.com/aws/aws-codebuild-docker-images/pull/519 

Wow we wasted a lot of time on figuring out what happened here. Couldn't you guys have scanned pipeline yaml files for go images and notified those folks?

Like # people like this

Does this break access for anyone using >= go 1.15?  We still have no access after the rollback?

Edit: Just read the update, it's been rolled out which explains why our pipelines broke.  No email on the 9th of May as suggested, other commenters suggest they also did not receive this either.  Obscure blog post to announce the breaking change, rollback for 1 week? then forward again.  

Like # people like this

Again this feature has been rolled out. This completely breaks any integration with AWS Lambda as it's still stuck on an unsupported Go version.

 

Please revert this as soon as possible again and hold on releasing as this makes using bitbucket repositories on AWS Lambda / CodeBuild impossible.

Like # people like this

Did you guys rolled back the  changes, Looks like its working now.

Or What I did is 

export GOPRIVATE="bitbucket.org/org-name"

We have a temporary work-a-round implemented in our code-build script.

 

We added the following in the install phase commands.

// TEMPORARY WORKAROUND to install latest minor fix of go
// remove after this merged https://github.com/aws/aws-codebuild-docker-images/pull/519
'wget -q https://go.dev/dl/go1.16.15.linux-amd64.tar.gz',
'tar -C /usr/local -xzf go1.16.15.linux-amd64.tar.gz',
'export PATH="/usr/local/go/bin:$PATH"',
"go version",

 

I recommend to keep using version 1.16 as you will otherwise run into problems running it in the lambda function.

Some of our pipelines are broken again.

We need to push AWS to this PR https://github.com/aws/aws-codebuild-docker-images/pull/519 so we can finally use standard (updated) codebuild image.

where is the announcement?

Like # people like this

I've had all of our Org Admins search their inboxes and there was no announcement sent to any of them regarding this issue.  How on earth were we expected to appropriately respond ?  This took our deployments down for over a business day while we quickly hacked together a fix.  Please provide a better story for how communications will be handled in the future.

Like # people like this

If Bitbuckets definition of "affected users" is noone then their comms were excellent.  We've searched for the email also but can't find it.  Pretty silent thread on Bitbuckets side here given the breaking change they introduced.  Where is the announcement for the second roll out?  They gave us an excuse to start our migration to Gitlab, one service done and tbh Gitlab is amazing so far.  It's significantly faster, has CI/CD built in, each repo has a docker registry.  Just wish we'd done it earlier.

Like Jayaraj Wilson likes this

Hi @Luke Choe ,

We have upgraded GoLang version to 1.17.11 in production. If you are planning to revert your changes, please make sure nothing breaks on this version.

Thanks,

Akhil

FYI AWS merged the upgrade for GO 1.15.15 and 1.16.15 https://github.com/aws/aws-codebuild-docker-images/issues/518 leaving my PR open for the next 1.17 and 1.18 https://github.com/aws/aws-codebuild-docker-images/pull/519/ (hoping they don't take more months of waiting)

This change seems to have broken some composer builds on our side as well - Specifically using Satis composer server.

The 404 status change means that we now get exceptions rather than login prompts or fall-back to SSH - this is baked into the core composer transport package, so not something we can fix quickly.

Completely unnecessary breaking change IMHO.

EDIT:: We've begun migrating to Gitlab. Breaking changes of this nature are entirely unacceptable. 

Like Andrew Jones likes this

Comment

Log in or Sign up to comment
TAGS

Atlassian Community Events