To improve performance, Bitbucket.org’s network 188.8.131.52/24 is anycast globally from 100+ 'edge locations' leveraging an AWS feature known as Global Accelerator - which we launched in October 2020 and announced on our blog.
Sometimes your network, or your Internet Service Provider’s network may have multiple, equal-cost paths towards Bitbucket to chose from: a concept known in networking as Equal Cost Multi Path (ECMP).
Almost always, a network device such as a router will forward all the packets within a single TCP Session across the same path and therefore to the same Bitbucket edge location. This is achieved via an algorithm known as 'Hash-based Load Balancing' (RFC2992) because the choice of path is consistently made based on a hash-function of the source and destination IP and ports - which are consistent during a TCP session.
A problem emerges if any part of your network or your Internet Service Provider’s network is equal-cost multi-pathing (ECMP) individual packets within a TCP session, rather than consistently multi-pathing entire TCP sessions via a hashing function.
In this scenario, different Bitbucket edge locations can inadvertently receive different parts of an existing TCP session. Because these edge locations will not have any knowledge of the TCP session, they will return a TCP Reset - which is an expected behavior and consistent with the appropriate IETF RFCs.
Anycast shift is an exceedingly rare problem. We have only seen it in 3 support cases among our many users.
The vast majority of issues involving TCP Resets are due to misconfigured Access Control Lists and Firewalls, which might be intentionally or inadvertantly configured to reject traffic to Bitbucket's IP addresses. We urge you to ensure HTTP(S) and Git/SSH ports are allowed to the IP addresses we publish here.
The second most common cause is TCP session failure due to Path MTU Discovery messaging being blocked by Firewalls and Routers, where a network tunnel of some form (e.g. a VPN) artificially reduces the Maximum Transmission Unit (MTU) of the TCP Session below the internet standard 1500 bytes without signalling the same to one of the parties. Cisco Systems has published a good article on this here.
Acknowledging that Anycast Shift is rare, some errors we have observed from Git and HTTP clients due to receiving a TCP Reset mid-operation are shown below as a guide.
These errors are most likely to be caused by other networking issues. They are not definitive signs that anycast shift is occurring.
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to bitbucket.org:443
git fetch returned status code 128:
stderr: fatal: unable to access 'https://bitbucket.org/': Network file descriptor is not connected
git fetch returned status code 128:
stderr: fatal: unable to access 'https://bitbucket.org/': TCP connection reset by peer
firstname.lastname@example.org' 'git-upload-pack ssh_exchange_identification: read: Connection reset by peer fatal: Could not read from remote repository.
curl -v https://bitbucket.org About to connect() to bitbucket.org port 443 (#0) Trying 184.108.40.206... Connected to bitbucket.org (220.127.116.11) port 443 (#0) Initializing NSS with certpath: sql:/etc/pki/nssdb CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none NSS error -5961 (PR_CONNECT_RESET_ERROR) TCP connection reset by peer Closing connection 0 curl: (35) TCP connection reset by peer
ssh email@example.com > /dev/null packet_write_wait: Connection to 18.104.22.168 port 22: Broken pipe
We advise you or your networking team (if you have one), and potentially also by working with your Internet Service Provider, to first identify any router which has more than one equal-cost path to Bitbucket.org's network 22.214.171.124/24. Linux and Unix tools like traceroute or mtr can be useful for identifying the device.
Once identified, a packet capture might be performed to discover if the device is indeed splitting packets within individual flows improperly. You, your networking team or your service provider, might then review product documentation or work with the device vendor to resolve the issue.
While this level of troubleshooting of customer and provider networks is beyond the scope of our support team, we are actively gathering more data on this issue, and would appreciate any insight you can share on the exact make, model and configuration of network device responsible, as this may help other customers in the future.
Please reach out to us via Bitbucket Cloud Support.
You can directly or via automation systems (like Docker, Salt, Ansible, etc) modify your operating system's hosts file to direct traffic to Bitbucket’s legacy IP addresses - which are not anycast to the internet:
126.96.36.199 188.8.131.52 184.108.40.206
This can be achieved by adding an entry to /etc/hosts on Linux and Unix based systems (including MacOS) like so:
sudo vi /etc/hosts # Add a line like so: 220.127.116.11 bitbucket.org # :wq to exit vi # or use any text editor of your choice
The final /etc/hosts will look like to this:
$ cat /etc/hosts ## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost 18.104.22.168 bitbucket.org
Atlassian has no plans to retire these IP addresses, but please be aware that ultimately the public DNS answer for Bitbucket.org is the authoritative IP address which clients should be using. Prolonged usage of a hosts override risks becoming out of date with the state of Bitbucket’s infrastructure and will not be as performant as our anycast edge.
Announced in this blog, this holiday season we’re celebrating all things CI/CD and between now and the end of 2019 we’ll be showcasing content, use cases, feature announcements and more. One featur...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events