Unknown Pipeline IP addresses

[Artur]

Hi guys, facing an issue today with the bitbucket pipeline,

it started coming from IPs that are not in your whitelists
(5 days ago was the last successful run and all worked fine)

So after debugging and allowing connections from any IP we discovered that the pipeline started working again but it came to us with the next two IPs to our server

(every run different IP)

44.195.38.14
52.91.75.69

We don't see these IPs in any of the known bitbucket whitelists.
Advice needed.
Thanks.

[Jaime Blasco · Accepted solution]

The solution is using an instance of type 4x or larger in the steps, which is outrageous. 

[Artur]

Wow, I see now, thanks for clearing this up!
We are using pipelines only for triggering,
so we don't use much power of pipeline instance.
P.S. Can you please point me to the pricing for 4x and above, I didn't find it.
Thanks in advance.

[rMcNeill]

Is there a source for this? Why do we suddenly need to change the resource size of pipelines to accommodate for their IP whitelisting, it seems worrying?

BTW - I tested and it works, but my questions is why the sudden change...

Thanks in advance.  

[Artur]

I found the answer to my question in another thread, here is the text:

"Cost is indeed based on the minutes used. The cost for 1x and 8x will be the same only if a pipeline runs eight times faster on a 8x instance, which is not always the case. If you pipeline is 50% faster on an 8x instead, the total cost will be 4x. Eg: - A pipeline take 14 minutes on a 1x instance. You will be charged for 14 minutes. - The same pipeline takes 7 minutes (50% faster) on a 8x instance. You will be charged 7 * 8 = 56 minutes (so, four times the cost of 14 minutes). When building applications, having more CPU isn't necessarily better because some compilers or unit-test frameworks can't make the most of N CPUs. I suggest you experiment with sizing to see what fits your pipeline the best. Perhaps the best cost/time ratio will be a 2x for a pipeline and 4x for another (also keep in mind that 2x will be migrated to the new architecture sometime in August)"

[Edmund Munday]

Hi @jaime@Artur @rMcNeill - PM for Bitbucket Pipelines here.

Wanted to share some context around these changes, and most importantly explain that these changes were implemented in order to actually avoid having to apply a non-trivial price increase for the Pipelines services to all customers.

I've provided a very detailed breakdown of exactly why this change has been made and the rationale for it here.

Fundamentally, it's a question of fairness for customers who do not use IP allow-listing (~90%), and not forcing them to pay more in order to subsidise the costs of those 10% who do use IP allow-listing.

All the details are on that post I linked, happy to answer questions you have and would greatly appreciate you posting them as responses to that linked post so that everyone can see the conversation.

[Krzysztof Sowa]

@Edmund Munday, thank you for sharing the article explaining the context around the changes.

We fall within the 10% of users who need to control the list of allowed IP addresses, so this change directly affects us.

Our use case for Pipelines is to trigger a deployment process on our server and wait for the process to complete while collecting some output. The resource usage on the Pipelines side is minimal, as its main task is simply to wait. However, because of this change, we’re now required to upgrade the environment to 4x capacity, incurring additional costs for resources we essentially waste.

As you can imagine I'm not the biggest fan of these changes. Since it complicates life of 10% of your users (or raises their bills), while reducing some of the >50% Pipelines costs (as mentioned in the article), I just hope the next step will be to lower prices for the remaining 90% of users.

[Edmund Munday]

Hi @Krzysztof Sowa - thanks for raising this point. This is actually something we plan to address for customers in situations like yours. We are planning to add the concept of a "Remote step" which should address this scenario.

The way they will work is basically like a Webhook, but backed by a step in the Pipeline. When the step starts, we will make a call to a configured URL and put the step into a "Pending" status. At some point in the future, the remote service will call back to Pipelines (using a short-lived auth token included in the initial request) and update the status of the Step, either marking it as failed or successful.

Once the status of the step is updated, the Pipeline will continue as normal. Obviously, you would not burn any build minutes while the remote step is waiting for a response as there is no actual container running in that context, we're just waiting for the async callback.

We're still in the early stages of planning with this, but we're hoping to make progress early in 2025.

[Krzysztof Sowa]

Thank you, @Edmund Munday for the heads up. I'll be looking forward for the feature you mentioned.

[Michael Klassen]

So ultimately, if you don't use the 4x/8x there is not going to be any whitelist ranges provided for the 10%?

[Edmund Munday]

Hi @Michael Klassen

Cross-posting this here for everyones visibility:

---

The documentation of Bitbucket Pipelines Cloud IP addresses is divided into two sections:

• Section 1: Valid IP addresses for Bitbucket Pipelines build environments
  
  This section applies to 1x/2x step sizes (or 4x/8x steps that have not been explicitly flagged to use atlassian-ip-ranges). An exhaustive list of IP addresses from which the traffic may originate on AWS can be obtained by using the following endpoint. You should filter records where the service equals EC2 or S3, and focus on the us-east-1 and us-west-2 regions. However, we do not recommend using these IP ranges as a security control due to their broad nature.

• Section 2: Atlassian IP Ranges
  
  This section pertains to steps specifically configured to use Atlassian IP ranges. These are applicable only to 4x and 8x size steps that have the atlassian-ip-ranges: true flag enabled. The step sizes 4x and 8x are only available for builds running under a paid Bitbucket Cloud plan (Standard or Premium)

To mitigate your issue and use the more restricted set of IPs, I suggest adjusting your pipeline configuration to include the following:

pipelines:
  default:
    - step:
       size: 4x
       runtime:
         cloud:
           atlassian-ip-ranges: true
       script:
          - echo "This step will use atlassian-ip-ranges"

I have highlighted in bold the options you need to add to ensure that your step uses the more targeted Atlassian IP ranges. You can find more details in the documentation here: Runtime Step Options.

I hope this helps resolve the issue you're facing. Please let us know if you need further assistance.

[Thomas Pötke]

Section 1 is not working , at least for standard plan users, since the pipeline has every time a different IP not in the new list. 

[Edmund Munday]

Hi @Thomas Pötke - just to confirm, the list of IP's on that page are for 4/8x steps when using the `atlassian-ip-ranges: true` flag in the .yaml.

If you are not using that setup, then your builds will come from an IP exposed via this endpoint.

[Thomas Pötke]

as standard plan user you have no choice, and for those, your provided list is not working. 

[Peter Fern]

This was absolutely terribly communicated, we had to find out the implications of "New Pipelines Runtime" by having all our deployments fail - parsing the documentation related to this is all but impossible, and there should have been very explicit communications delivered to all customers about the impending build breakages, including specifically what options are available to mitigate them (possibly none, other than allowing all of AWS).

Also, I'm baffled that Atlassian thinks it can estimate how many users are affected by this, since it's impossible for them to know how many customers actually have IP restrictions in place.

[sdrb]

It's a 'hidden' price increase.

Switch to 4x builds (4x price) or turn of your firewalls...

[ozkantastan]

its unacceptible!!!!

[Edmund Munday · Accepted solution]

Hi all.

• A few people have raised concerns regarding how these changes were communicated which we wanted to clear up here.
• Breaking changes are never a desired outcome - to manage this we have very detailed and standardised deprecation notice policies and processes which we take very seriously.
• In line with these processes:
  • Six months notice was provided to all users of the product via a range of different communication channels.
  • This notice was then repeated 3-months out from the change.
  • 2 weeks out from the change.
  • And (finally) during the change implementation itself.
• These comms channels include:
  • Multiple public blog posts on the Bitbucket product blog.
  • Multiple posts to the in-product "What's new" product update feed.
  • Multiple in-product popup prompts that were seen by every user of the product and included call-to-actions such as "Changes to Pipelines IP Addresses"
  • And multiple posts on both the Bitbucket and Pipelines community spaces.
• We understand that sometimes, people do just miss these kinds of notifications/communications, and we have implemented a range of processes to provide extensions for customers who missed these comms and needed more time to complete the required changes.

[bradley.carter]

Experiencing all of the above issues. My production pipelines dropped recently because of this.


@Artur / @Syahrul we will be moving over to self-hosted pipelines with internal static IPs to prevent this becoming another future issue, the totalled build-minute cost is now outweighing the self hosted cost on a basic EC2 instance. Where we can host ourselves for a fraction of the price.

Forcing us to move to 4x instances and not providing a static IP address range for 1-2x is not a great solution for this 10%.

Brad

[bradley.carter]

Frankly, you are about to see a mass exodus.

[Edmund Munday]

Hi @bradley.carter - thanks for sharing your feedback on these changes.

I completely appreciate how this may be frustrating, but it's important for people to understand the context surrounding these changes which has been shared by myself here.

Most importantly, it's critical for customers to understand that, even after these changes, Bitbucket Pipelines is still the only mainstream SCM + CI/CD provider in the market that even has the option of static IP's on a managed-SaaS tier CI/CD offering.

Neither of the other major vendors in this space offer static IP's on any of their managed-SaaS tiers, let alone all of them.

We understand some of these changes are not ideal, but I'm very confident in saying that we are doing absolutely everything possible to ensure the product remains flexible for our customers, is priced fairly relative to each customers usage-pattern and needs, and remains sustainable over the long-term.

[Leonardo Rojas]

> Forcing us to move to 4x instances and not providing a static IP address range for 1-2x is not a great solution

you share my pain

[Edmund Munday]

Hi @Leonardo Rojas - Would be great to get some insights into what workloads you are doing that require a static IP with a 1/2x step.

[Leonardo Rojas]

Hi @Edmund Munday  - I already resolved the issue that I was getting, I had to add the `options` section before the `pipelines` section:

It took me a long while (hours) to figure out what was going on, reading, testing and understanding the situation, figuring out which options I had to fix all our broken pipelines :(.
We'll later see how much this change will increase the bill due to 4x...

[Edmund Munday]

Hi @Leonardo Rojas - that will apply the 4x sizing to all steps in your Pipelines.

- If that is what you are wanting to do, then that's great.

- If you only want to apply 4x to specific steps in your pipeline (e.g. a deployment step), you can apply that option at the step level as per below.

- step:
   size: 4x
   runtime:
     cloud:
       atlassian-ip-ranges: true

[Marco Aurélio]

Shitty solution! Congratulations bitbucket team for making what was already bad even worse! Hopefully I can convince the manager to move to Github or Gitlab!

[bradley.carter]

@Edmund Munday 

You must understand. This 10% contains your biggest contractual clients. Given we operate with a security standard that requires static IPs but at lowest possible cost.

If we can achieve the same security standard by moving to self-hosted, Jenkins or Gitlab etc, we will do it. We are not obligated to stay.

Enforcing an immediate blockage in connectivity PLUS a 2-4x cost increase for just a pipeline runner is simply unheard of.

Please let us know if you plan on reverting or at least appeasing this.

[yeasir]

@bradley.carter : Were you able to migrate to self hosted runners. If your runner is behind a corportae firewall, how would you allow bitbucket to communicate to your runner to start the pipeline? Just wondering if that approach will end up looping into the same hundreds of IP whitelisting issue. Thanks. Yeasir

[Krzysztof Sowa]

@yeasir the self-hosted runners work slightly different. It's the other way around, actually. On your server you run a docker container, which connects to Bitbucket, so there's no need to allow incoming connections at all. The setup process is rather straightforward. However we found it slower (or more expensive) than using Pipelines, especially with multiple parallel steps and/or multiple pipelines running in the same time.

[bradley.carter]

@yeasir we were able to for a limited time. Then back to square one. Moved to another source control.

[bradley.carter]

@Edmund Munday given the pushback here, are you able to grace the rest with any updates? Was it a management mistake? More explicitly, do you have plans to change this decision? Pls reply.

[Edmund Munday]

@bradley.carter

As explained, Pipelines is currently the only major SaaS SCM + CI/CD vendor that even offers static-IP's on it's managed SaaS tiers.

All other vendors either do not offer static IP's at all, or require you to upgrade to an enterprise tier plan + provision managed runner agents with static IP's.

We fully appreciate that the ideal scenario would be to offer static IP's to every customer at zero additional cost, but unfortunately that is simply not feasible.

• I do want to stress that we have put in significant amounts of effort to ensure that we do still offer static IP's on SaaS tier builds, unlike all other major vendors.
• Additionally, you are only required to use those larger sizes on steps that are communicating with your infrastructure - which will be only a small sub-set of your overall build.

Regarding potential changes to the updated structure:

• The only realistic addition that could be made in order to give customers more flexibility would be allowing customers to opt-in to using the `atlassian-ip-ranges` flag on smaller-sized steps and applying a price multiplier to those steps to account for the significantly increased cost of providing the static IP addresses.
• For example (this is purely hypothetical, just for the sake of conversation):
  • A 1x step + `atlassian-ip-ranges` flag may be billed with a +100% price modifier.
  • A 2x step + `atlassian-ip-ranges` flag may be billed with a +50% price modifier.
• The reason this was not included initially was that we did not want to add complexity to the logic that determines step pricing so that there was no risk of customers inadvertently using a configuration that would cost them more without them realising.
  • Additionally, especially for 2x steps, the increased price after the multiplier would be so close to the price of a 4x step that it would likely end up cheaper for customers to use a 4x step, gain the benefits of the increased CPU capacity to shorten the amount of time the step runs for, and therefore incur a lower charge.

We'd be keen to hear people's feedback/thoughts on this potential path.

---

In regards to comms:

• Breaking changes are never a desired outcome - to manage this we have very detailed and standardised deprecation notice policies and processes which we take very seriously.
• In line with these processes:
  • Six months notice was provided to all users of the product via a range of different communication channels.
  • This notice was then repeated 3-months out from the change.
  • 2 weeks out from the change.
  • And (finally) during the change implementation itself.
• These comms channels include:
  • Multiple public blog posts on the Bitbucket product blog.
  • Multiple posts to the in-product "What's new" product update feed.
  • Multiple in-product popup prompts that were seen by every user of the product and included call-to-actions such as "Changes to Pipelines IP Addresses"
  • And multiple posts on both the Bitbucket and Pipelines community spaces.
• We understand that sometimes, people do just miss these kinds of notifications/communications, and we have implemented a range of processes to provide extensions for customers who missed these comms and needed more time to complete the required changes.

[Raul Jimenez]

Same here

This is annoying and worrying

[Edmund Munday]

Hi @Raul Jimenez - PM for Bitbucket Pipelines here.

Wanted to share some context around these changes, and most importantly explain that these changes were implemented in order to actually avoid having to apply a non-trivial price increase for the Pipelines services to all customers.

I've provided a very detailed breakdown of exactly why this change has been made and the rationale for it here.

Fundamentally, it's a question of fairness for customers who do not use IP allow-listing (~90%), and not forcing them to pay more in order to subsidise the costs of those 10% who do use IP allow-listing.

All the details are on that post I linked, happy to answer questions you have and would greatly appreciate you posting them as responses to that linked post so that everyone can see the conversation.

[Finererp]

if we are having 100 yaml file, we have to make changes in all the 100 files. That pretty nice.  You guys have a big team, you should think atleast about customers.

[Marco Aurélio]

Bitbucket you suck!

[Leonardo Rojas]

I could not find more expressive words.

one does not simply apply breaking changes on production...

[Matthias Maier]

@Edmund Munday it seems that we are also one of the 10% customer which are using IP allow-listing.
We use IP allow-listing for our deployments to AWS EKS for security reasons (because we don't want that the whole world can access the kubectl API of the kubernetes cluster).

Currently the pipeline is just size 1x because we use the pipeline to trigger the helm chart deployment to kubernetes, so the pipeline itself doesn't need a lot of resources.

I already verified the pipeline config snippet you provided to use the atlassian IP ranges and it works well, except of the fact that we now pay 4x times more money for each deployment :(

[Edmund Munday]

Hi @Matthias Maier - thanks for sharing this. The situation you mentioned is one that we are looking into potential mitigations for, but I don't have anything I can share right now.

The challenge is finding a solution that is not open to abuse (you would not believe the amount of abuse-cases we deal with running a platform like Pipelines!), and allows us to keep control of the amount of data that passes through those static IP's (as it's data transfer that makes costs go through the roof).

As always, we are keen to hear suggestions and feedback from people like yourself. I will be monitoring these community posts myself, so feel free to share anything you would like.

[Jaime Blasco]

Add this in you pipeline config

options: runtime:
    cloud:
        atlassian-ip-ranges: true

pipelines: default:
- step: size: 4x

[Volodymyr Kukurudz]

I tried this and still have the same issue. Could you send screenshot/example of how it looks please? 

[Matthias Maier]

@Volodymyr Kukurudz you can apply the pipeline config for atlassian-ip-ranges directly to a step

this one worked for me:

- step: &deploy-pipeline
    name: Deploy
    size: 4x
    runtime:
      cloud:
        atlassian-ip-ranges: true
    script:
      - echo "start deployment..."

[Thomas Pötke]

another 10% customer here. Annoying and one more reason to migrate to another product. 

Still no fix for https://jira.atlassian.com/browse/BCLOUD-18261 but increasing the plans ....

[Edmund Munday]

Hi @Thomas Pötke completely understand where you're coming from.

Regarding https://jira.atlassian.com/browse/BCLOUD-18261 - That feature was shipped ~18 months ago. What specific element of that are you having issues with?

---

Regarding these IP changes:

Fundamentally, this decision was made for the benefit of the 90% of customers who do not use static IP's in their builds. With static IP's making up over half the operating costs of Pipelines before this change, we felt it was fundamentally unfair for 90% of customers who do not use static IP's to be massively subsidising the costs incurred by the 10% that do use static IP's.

The alternative would have been an across-the-board price increase for all customers.

The decision process around this change is one we've been very deliberate and transparent about, announcing these changes several months in advance along with publishing a detailed breakdown of the logic behind the decisions.

I do also want to take this opportunity to also call out that Bitbucket Cloud is still the only mainstream combined SCM + CI/CD provider to offer static IP's on any of their managed SaaS CI/CD tiers. All other vendors in this space have moved to a floating IP only structure for their managed SaaS tiers.

A good summary of that information is available on the post I published here.

[Syahrul]

Hi Everyone,

We have recently updated our 1x/2x size option builds to operate from a new, broader IP range.

If you need your pipeline builds to operate from a more limited set of IP addresses, consider using the atlassian-ip-ranges configuration available with our 4x/8x steps. This setup offers better security by narrowing down the IP addresses used. You can find more information about this configuration here.

To provide some extra context on this change and the requirement for 4x/8x steps to access restricted IP's, please see the discussion re: this topic from the Pipelines community space here.

Please Note: Opting for larger step sizes (4x/8x) may impact billing. We encourage you to review the relevant documentation on step sizes here to understand these implications fully.

If the 4x/8x step sizes are not suitable for you, you can use 1x/2x and access the full list of IP addresses in AWS JSON format. This list can be filtered specifically for EC2 or S3 resources in the us-east-1 and us-west-2 regions.

Note: The IP addresses listed are managed by Amazon and are subject to change. Therefore, relying solely on IP-based firewalls for securing your infrastructure is not recommended. Instead, consider implementing secure authentication methods for any services exposed to Bitbucket Pipelines, such as using OpenID Connect (OIDC). Allowlisting the entire range of AWS IPs can potentially expose your system to risks.

Having said that, we suggest using the 4x/8x runtime option. By choosing the 4x/8x runtime, you benefit from increased performance and a more controlled network environment. This makes it an excellent choice if you want to prioritize security and efficiency in your CI/CD processes.

Regards,
Syahrul

[Matthias Maier]

Using 4x/8x pipeline steps to enable atlassian IP ranges for a script that only triggers a deployment to kubernetes without using much resources isn't an excellent choice to prioritize security. It is rather a new requirement by Atlassian to enable fixed IP ranges and causes an increased bill due to 4x/8x charged build minutes. IMO there is only a benefit for Atlassian.

[Edmund Munday]

Hi @Matthias Maier - completely understand where you're coming from.

Fundamentally, this decision was made for the benefit of the 90% of customers who do not use static IP's in their builds. With static IP's making up over half the operating costs of Pipelines before this change, we felt it was fundamentally unfair for 90% of customers who do not use static IP's to be massively subsidising the costs incurred by the 10% that do use static IP's.

The alternative would have been an across-the-board price increase for all customers.

The decision process around this change is one we've been very deliberate and transparent about, announcing these changes several months in advance along with publishing a detailed breakdown of the logic behind the decisions.

I do also want to take this opportunity to also call out that Bitbucket Cloud is still the only mainstream combined SCM + CI/CD provider to offer static IP's on any of their managed SaaS CI/CD tiers. All other vendors in this space have moved to a floating IP only structure for their managed SaaS tiers.

A good summary of that information is available on the post I published here.

[yeasir]

Hi @Edmund Munday

Another person from a tech team affected by this change.

Re `Fundamentally, this decision was made for the benefit of the 90% of customers who do not use static IP's in their builds`:  I did not understand this part. We do not use any fixed ips in our build. Our server are hosted in aws. The pipeline runs some tests and then connects (via ssh) to  our aws servers to run some commands for the deployment. But from the last 3 days ssh is failing. We use aws security policy to allow/deny access to our server, along with other security measures. If our build is not dependant on any specific ips, why am i being impacted? Or did i read your statement wrong? Thanks

[Edmund Munday]

Hi @yeasir - is AWS blocking access using a set of allow-listed IP addresses?

If so, that is what would be causing the issue.