Deployments Failing all of a Sudden, Connection Refused - IP Whitelist Issue

[Alexander Krause]

My Bitbucket Pipeline Deployments have all started failing consistently as of the start of this week.

Error message:

ssh: connect to host [redacted] port 22: Connection refused

rsync: connection unexpectedly closed (0 bytes received so far) [sender]

rsync error: unexplained error (code 255) at io.c(231) [sender=3.3.0]

I was able to push code without issue last week, but for some reason, the connections are all now refused when the rsync image I am using (https://bitbucket.org/atlassian/rsync-deply) attempts to move the code to my server.

I maintain several sites at a university, and they are all on university-maintained VMs, which are protected by a firewall that requires all incoming and outgoing connections to be whitelisted.

I have been using Bitbucket Pipelines for about 2 years now, without any issues. The list of IPs I have whitelisted, according to the Technical Specification Document I have from my university server admins, is the following.

3.216.235.48

3.221.151.112

34.199.54.113

34.216.18.129

34.218.156.209

34.218.168.212

34.231.96.243

34.232.25.90

34.232.119.183

34.236.25.177

35.155.178.254

35.160.177.10

35.171.175.212

44.199.3.254

44.199.45.64

44.199.127.226

52.41.219.63

52.54.90.98

52.72.137.240

52.202.195.162

52.203.14.55

52.204.96.37

52.205.184.192

174.129.205.191

This list has served me well over the last two years, up until Tuesday of this week when I tried to push up a simple bugfix to one of my sites, only for the deployment step to fail. Since then, I have not been able to connect to any of my servers via the cloud runner machines that handle this step.

I checked with my server admin, and they assured me that no changes had been made on their end. I also shared with them the documentation of “IP addresses and domains to allowlist in your corporate firewall” (https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/).

After reviewing this, my server admin got back to me with the following:

“The IP addresses listed in the Bitbucket document are included in your TSD as being able to have access via port 22. I compared the list directly and presuming the list at Bitbucket is up to date, these should all still be there.”

I also shared with him this blog post, which seemed relevant. https://bitbucket.org/blog/ip-address-update-2024

Most notable, is the “Firewall considerations” section. However it appears that that section refers to outbound connections, from my server to Bitbucket, but the issue is with inbound connections, from Bitbucket to my server(s).

I then found this page on the Atlassian site: https://confluence.atlassian.com/bbkb/troubleshooting-bitbucket-cloud-pipeline-ip-whitelisting-1209869206.html

I added the "curl ifconfig.me" command to my deployment step, which is the one that is failing.

I ran the failing step a few times and documented the resulting IPs and times, which resulted in this list of IP addresses that attempted to connect to my server.

54.147.109.37

3.81.17.66

44.202.51.50

98.84.41.228

75.101.235.132

54.147.109.37

54.147.109.37

54.147.109.37

54.87.80.152

3.88.28.240

3.88.28.240

It appears that these are all using AWS, which I suspect means that the IP list could be literally anything at this point.

I don’t usually post on forums like this asking for help, but as of now, I don’t have a paid plan with Bitbucket, so I cannot contact support directly.

I am exploring getting the paid plan, but I have to go through the contracts department at my university to get approval before I can do so, and that could take quite some time.

So, if anyone could provide me with any insights into why this is occurring all of a sudden, I would be forever grateful.

I am happy to provide any other information that may be needed in order to get some clarification on what has caused this issue, and what steps can be taken to resolve it.

My hope is that some additional IP addresses just need to be whitelisted and then this will all start working again.  If that is the case, can someone point me in the right direction to where I can get this list?

Thank you in advance!

[Jim Knepley - ReleaseTEAM]

Bitbucket Cloud is hosted on AWS, so it follows that the IP addresses are from AWS blocks. I don't think I would bet on those IP addresses being static, but the network blocks they are part of are probably more consistent.

AWS publishes a list of their IP ranges. You can narrow it down by region easily enough and might be able to narrow it down by AWS service.

However...

It's curious, the error is "connection refused" which is a TCP message that means that there isn't a service listening on that port number. When there's a firewall in the way, the response is more commonly a timeout, at most you'll get an ICMP "administratively prohibited" back (and usually not even that).

[Alexander Krause]

Thanks for the response!

I suspect that the approvers on the server admin side that approve external connections to the machines at my university will likely not approve such a broad range of IPs from AWS, even if we narrow it down substantially, especially given the fact that these machines will be pushing files directly to the servers.

I am wondering exactly what changed on the Bitbucket side of things that made this all of a sudden become an issue as of this week. I know there are blog posts about recent changes to IPs, but it seems like my static list worked without issue for a long time, and now all of a sudden this has become a problem that I don't seem to have a clear solution to.

Are there any other suggestions you can make that could help resolve this, and do you have any additional insight into the changes that recently occurred on the Bitbucket end that could be the cause of this issue?

Also, regarding the connection refused errors I am seeing. I checked with my server admin and they said that it is pretty standard for our firewall to return a connection refused error such as what I am seeing when the connection is not on the whitelist.

Any assistance or sight that you can provide would be greatly appreciated. I really don't want to have to return to pushing files up manually to my servers, but without access to these runner machines, I am not sure what other choice I have. I could potentially create my own runner machine, but there are resource concerns with setting up and maintaining such a machine, so the cloud-based runner machines that Bitbucket provides are preferred.

[Jim Knepley - ReleaseTEAM]

I don't have special insight into any of Atlassian's changes. 

A layer 4 firewall can send a TCP RST in response to a blocked connection attempt, but in my experience, it's much more common for blocked traffic to just be dropped.

Perhaps don't push the files directly to servers... your build process could create an appropriate package file, store it somewhere, and have the servers pull it down with something like Chef/Puppet/Ansible.

[Alexander Krause]

That all makes sense. I will have to look into your suggestion. I don't have much experience with Chef/Puppet/Ansible. Again I do appreciate your time and knowledge on this subject.

Alternatively, I believe I may have discovered another possible solution that would require me to get the paid plan (which I just got approval to purchase).

From this page from Atlassian Support: https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/#Valid-IP-addresses-for-Bitbucket-Pipelines-build-environments

From the "Valid IP addresses for Bitbucket Pipelines build environments" section and the following section "Atlassian IP ranges".

It looks like, if I set the size to 4x or 8x, which I believe requires at least a Standard paid plan, then I can use the smaller sub-set of IP addresses, which I suspect would be much more likely to get approved by the powers that be at my institution.

I have run this by my server admin and am waiting to hear back.

Since I am already approved to purchase that Standard plan, I think this may be a viable solution, depending on what I hear from the server management end of things.

So with that being the case, I am going to explore that route, and if it doesn't work out, I will then consider your suggestion, which again, I greatly appreciate!

If you have any additional insight to share regarding the above, I am happy to hear it.

Thank you again so much for your time and attention!

[Jim Knepley - ReleaseTEAM]

In my experience, if you present your admins with a link to ranges like what you referenced, they're much more likely to implement it because they can't argue with the vendor.

Good luck.

[Alexander Krause]

Thanks again for all your help here!

For the sake of completeness, I wanted to give an update.

I purchased the Standard plan this morning and updated my bitbucket-pipeline.yml file to use the 4x size and the atlassian-ip-ranges for the deployment steps, and everything is now working as expected.

I actually didn't even need to get any new IP addresses approved. The list I already had approved was identical to the one used for the 4x and 8x sizes.

[remon]

Can you show how you did that? I don't seem to have any luck with using a size: 4x ...

[remon]

options:
  size: 4x
  runtime:
    cloud:
      atlassian-ip-ranges: true

but, that also comes with a cost, on which Atlassian is quite vague, but coworker found notice that a 4x instance uses build minutes at 4x rate. So, 5 minute build is billed as 20 minutes.