Sharing Some Context On Our New Pipelines Step Sizes

[#1]

Hi all,

Since releasing our new Pipelines Runtime we've received a huge amount of positive feedback, widespread adoption, and also a range of questions relating to some of the more low-level details of how the new runtime works and what the plans are for it over the next few months.

One specific question that has been raised a few times is relating to IP Allow Listing and our 1/2x step sizes. In particular, people wanting to understand the reasoning behind restricting the `atlassian-ip-ranges` capability to 4/8x step sizes. 

In light of these questions, I'd like to share some context around the decisions so everyone can understand the thinking.

---

Fundamentally, the decision to restrict the `atlassian-ip-ranges` to 4x/8x steps comes down to a function of "cost" & "customer fairness" (for lack of a better word).

You may/may-not be aware that over the last few years, cloud infrastructure vendors have changed the way they charge for services, and one area in particular that has impacted is network ingress/egress. To put it bluntly, the cost per MB of traffic via a static-IP has grown exponentially over the last few years, to the point that >50% of the cost of running Pipelines as a service was being consumed by the network traffic via those static IP addresses alone.

Where this gets more complicated is the fact that only a tiny minority of customers actually leverage IP Allow-Listing, so essentially a huge portion of the cost of running the service was being spread across every single customer, but the benefit that cost was paying for was only being gained by a very small minority (<10%) - something we believe is not fair from a customers point of view.

Given the rapidly rising cost of providing static IP's as standard, we were left with effectively two options:

1. Keep static IP's as a standard feature for all builds and implementing a non-trivial cost increase for all build sizes to cover the cost of the static IP's.
   1. noting that competitor SaaS CI/CD solutions do not include bundled static IP's on any builds, let alone for all builds
2. Restricting static IP addresses to larger step sizes, allowing us to more feasibly absorb the increased cost of providing them.

Removing static IP's for 1/2x steps allows us to continue offering those step-sizes at the price-point they are currently being offered at, the alternative would have been a pretty significant price increase, which we obviously wanted to avoid.

4/8x steps are, by nature, much more heavily utilised by customers at the more "mature" end of the DevOps spectrum, and we are expecting there to be a much higher overlap between customers implementing 4/8x steps, and customers using IP allow-listing, reducing the imbalance of value being provided to customers who are/aren't using that feature.

---

A few notes to end on:

• We did consider the option of offering some kind of "1/2x premium" which would include the ability to run those steps with static IP's for an incrementally higher cost.
  • However, once we ran the numbers on what those steps would have to be priced at in order to cover the significantly higher network costs, it didn't make sense to offer that option at all.
  • The pricing that would be required for those steps to be commercially sustainable would have put them within a stone-throw of the price of a 4x step anyway, making the whole offering redundant.
• The floating IP model being moved to for our 1/2x steps is the same model already used by all other Tier 1 SCM + CI/CD providers for their SaaS step sizes.
  • Even after this change, we will still be the only service in our space that provide any way to implement static IP's for builds without having to spin up your own runners.
  • All other equivalent providers operate exclusively off dynamic IP ranges for their SaaS tier CI/CD services.
  • We are very aware of the importance of this functionality from a security & compliance perspective, and we have put in significant effort to continue to make this available to customers quickly and easily, without requiring them to do things like spin up dedicated runners.

---

How were these changes communicated:

• A few people have raised concerns regarding how these changes were communicated which we wanted to clear up here.
• These changes were communicated with six months, 3 months, and 2 weeks notice to all customers of Bitbucket Cloud via a range of different comms channels.
• These comms channels include:
  • Multiple public blog posts on the Bitbucket product blog.
  • Multiple posts to the in-product "What's new" product update feed.
  • Multiple in-product popup prompts that were seen by every user of the product.
  • And multiple posts on both the Bitbucket and Pipelines community spaces.
• We understand that that sometimes, people do just miss these kinds of notifications/communications, and we have implemented a range of processes to provide extensions for customers who missed these comms and needed more time to complete the required changes.

[sdrb]

Whatever the rational may be for making a change like this. This has been poorly communicated.

The tl:dr; is: "we are increasing our pricing". Unhappy - premium customer

[Boris]

Listen, your decision broke my staging and production builds. My developers couldn't deploy for a full day and I had to spend the last few hours first figuring out whether I am missing some IP ranges from my firewall, then figuring out what IP the deploy started to use, and then trying to search through the various messages in this forum to figure out how to actually implement atlassian-ip-ranges: true in my pipelines.

You could have raised prices and offered a cheaper option for people that opt out. That would have been the more professional approach. Instead you decided to introduce breaking changes that were poorly communicated.

[Lincoln Russell]

I'm in the same position as Boris — you broke my production pipeline and wasted days of engineering hours, which are more expensive than your product. Breaking changes are worse than price increases, always. Static IPs are a standard feature of CI pipelines and our pipeline breaking mid-week is on you, context & justifications be damned.

[Josh Williams]

This all kind of makes sense (at least for on-prem whitelisting which Atlassian has no control over), except this:

YOUR OWN IP whitelist feature in BB is blocking your own BB Pipeline AWS servers.

Make that make sense.

We make BB API calls in our steps to get commits, but the server you give me in BitBucket pipelines is blocked from calling BitBucket API? 

You would think Atlassian's IP Whitelist feature would implicitly (without us having to enter 400 IPs manually) whitelist calls coming from their own pipelines servers to their own Bitbucket servers.

Or at least give us a checkbox in the IP Whitelist Settings: Allow BB Pipeline Servers Access

[Ryan Leach]

for everyone else that ends up here as a result of a search or support ticket:

I just want to note officially that you are being strong-armed into either:

paying more for 4x/8x, even if it’s an underutilization of those resources, and means potentially making a change to ALL your pipelines

Switching to runners, which you would have to put operational and financial capital into maintain, as well as potentially changing ALL your pipeline

compromising your security be removing any IP source validation in the trust policy of your OIDC roles and connections

I agree this was poorly communicated and I lost a day to working out the security implications and a lot of unhappy devs and product owners

[Edmund Munday]

Hi all.

• A few people have raised concerns regarding how these changes were communicated which we wanted to clear up here.
• These changes were communicated with six months, 3 months, and 2 weeks notice to all customers of Bitbucket Cloud via a range of different comms channels.
• These comms channels include:
  • Multiple public blog posts on the Bitbucket product blog.
  • Multiple posts to the in-product "What's new" product update feed.
  • Multiple in-product popup prompts that were seen by every user of the product.
  • And multiple posts on both the Bitbucket and Pipelines community spaces.
• We understand that that sometimes, people do just miss these kinds of notifications/communications, and we have implemented a range of processes to provide extensions for customers who missed these comms and needed more time to complete the required changes.

In regards to the breaking nature of these changes, and questions around why this was not implemented as a price increase retaining existing behaviour as the default.

As explained in the initial post, coving the rapidly increasing costs associated with static IP's would have illicit a price increase in the area of 2x the baseline. After very extensive review of the data available to us we made the decision that imposing a 2x price increase on all customers, when the feature triggering that increase was only used by less than 10% of customers, was fundamentally not the right thing to do and would have been borderline unethical given the commercial implications of such a change.

We understand breaking changes are never a desired outcome, to mitigate this Atlassian has very detailed and standardised deprecation notice policies and processes which we take very seriously.

In line with these processes, six months notice was provided to all users of the product via a range of different communication channels. This notice was then repeated 3-months out from the change, 2 weeks out from the change, and (finally) during the change implementation itself.