Bitbucket Cloud has landed in AWS

[#1]

This blog was written by Bitbucket's Head of Engineering, Daniel Tao, and is also published to the Bitbucket Cloud blog.

In July and August of this year, I revealed to our community for the first time that Bitbucket Cloud was engaged in a project to migrate to a new platform. As I wrote in July:

For over a decade, the majority of Bitbucket’s services have been hosted in a data center. While this has served us well for many years, operating a data center comes with significant overhead as well as risk. For example, when we have had unexpected capacity issues (e.g. hardware failures or unplanned outages in upstream services), we have been limited by the physical servers we had available, impacting our time to recovery.

Over the past year, we have been on a journey to migrate all of Bitbucket Cloud to Micros, Atlassian’s internal cloud platform based on AWS. This is truly a quantum leap for Bitbucket Cloud and will resolve many reliability issues including the one described above and more.

I am thrilled to announce that, in late August, we officially completed this project and Bitbucket Cloud has been operating in AWS, data center-free for over 2 months.

So how does this benefit our customers?

1. Reliability. Bitbucket’s infrastructure has never been more resilient and scalable.

   1. We have had zero major reliability incidents (i.e. outages or degraded performance) since the completion of the migration.

   2. Over the past two months since completing the migration, our volume of weekly support cases related to reliability has decreased 93%.

2. Security. Bitbucket is more secure today than ever before.

   1. Our cloud storage volumes providing our web services access to customer source code are now encrypted at rest.

   2. Running on Atlassian’s cloud platform (Micros) allows us to leverage security best practices and policies enforced platform-wide, such as resource isolation via security groups, regular OS scanning for known vulnerabilities, regular data backups, and more.

   3. Removing data center operations from our teams' responsibilities eliminates an entire class of security risk: servers, drives, network interfaces, and other hardware that needed to be regularly updated and protected.

3. Performance. Thanks to major investments in Bitbucket’s architecture to reduce our services' sensitivity to file system latency, our web response times across our core capabilities today are on average 55% faster than back at the start of 2020.

This is one of the most technically complex projects we’ve ever tackled as a team. The following are just a few stats to convey the scale of this accomplishment:

1. We migrated over 50 million repositories to our new cloud storage backend.

2. Additionally, we replicated and moved all other data stores with traffic totaling over 200,000 queries per second to AWS.

3. Our services regularly handle over a billion daily transactions which we redirected transparently to elastic compute resources in AWS with zero downtime.

I couldn’t be prouder of the many engineers—both within the Bitbucket Cloud team and across the multiple Atlassian platform teams who supported us—who made this migration possible. On that note, stay tuned for a more in-depth engineering blog post in the coming weeks wherein we will share some highlights covering the work we did to achieve this, challenges we faced, and lessons learned along the way.

[Bastian Stehmann]

Hi @dtao ,

Thanks a lot for this update and for all your efforts to make bitbucket more stable and reliable. 

[Simon Logan]

Hi @dtao 

Does this mean that customers have some control over which AWS region their data is hosted in?

If not at the moment, is that something that's planned?

Thanks,

Simon

[Simon Logan]

Hi @dtao / @Bastian Stehmann ,

Can I get a response to my query from a couple of weeks ago please?

Thanks

Simon

[Bastian Stehmann]

Hi @Simon Logan ,

Currently there is no way to choose the AWS Region.

If I remember correct, it's US West by default and US East as backup. 

[Simon Logan]

Thanks for the update @Bastian Stehmann , do you know if there are any plans to host in London region on the roadmap?

I think it would be something a lot of potential customers would be keen on.

Thanks ,

Simon

[dtao]

@Simon Logan sorry for not responding sooner! @Bastian Stehmann is correct that there is currently no way to select a region; but it's US East by default with US West as backup (opposite of what he said).

The good news is that yes, we do have plans to operate in additional regions in the future. Data Residency is a project on our roadmap tentatively scheduled to be delivered in the next 1-2 years. The Bitbucket Cloud and Bitbucket Data Center teams are currently collaborating on a next-generation storage technology for Git repositories that will serve as the foundation for this project, which you can expect to see on our roadmap at some point in the future, as we get closer to having something ready for production.

[Simon Logan]

That's good to know @dtao , I'll keep an eye out for it.

All the best.