We cannot access our instance neither sandbox since late Monday EST.
Any updates besides the ones showing on the Atlassian status page?
Thanks,
Olimpia Estela
Hi Everyone,
In an Easter miracle of Biblical proportions, our site has been resurrected!
Here is what Atlassian support have sent us:
We have successfully completed site restoration! Please follow the steps below to get started.Step 1: Validate your sitePlease validate your site and respond to this ticket with any outstanding issues or concerns and we’ll make it our priority to work through them with you. Our engineering and support teams are standing by to address any additional issues.Step 2: Enable mailNote that your incoming and/or outgoing mail will be disabled at this point. Once you have thoroughly validated the site, you can enable it by going to the link below:Outgoing mail - https://<YourSite>.atlassian.net/secure/admin/OutgoingMailServers.jspaIncoming mail - https://<YourSite>.atlassian.net/secure/admin/GlobalMailSettings.jspaIf you would prefer us to turn this functionality back on for you, please reach out to us via this support ticket.Step 3: Sync groupsIf you have user provisioning configured at your Organization you might need to resolve potential conflicts with syncing groups to ensure the changes from your Identity Provider during the outage get reflected in your site. To achieve that, please follow the instructions below:Access https://admin.atlassian.com/Select your organizationNavigate to Directory > User provisioning. If you don't see User provisioning under the Directory tab, go to Settings and it should be there.You might see a banner asking you to resolve the syncing conflicts. Banner will show: <group count> groups pending sync(If you don't see a banner, no action is needed.)Click on ‘Review groups before sync’.Expand the groups to see what will change (who will be added/removed from that group). You can always add/remove users later at the Identity Provider administration dashboard.This process will link these groups to the ones at your site(s) and allow you to add/remove people based on what we receive from your Identity Provider.
We have successfully completed site restoration! Please follow the steps below to get started.
Step 1: Validate your site
Please validate your site and respond to this ticket with any outstanding issues or concerns and we’ll make it our priority to work through them with you. Our engineering and support teams are standing by to address any additional issues.
Step 2: Enable mail
Note that your incoming and/or outgoing mail will be disabled at this point. Once you have thoroughly validated the site, you can enable it by going to the link below:
Outgoing mail - https://<YourSite>.atlassian.net/secure/admin/OutgoingMailServers.jspaIncoming mail - https://<YourSite>.atlassian.net/secure/admin/GlobalMailSettings.jspa
If you would prefer us to turn this functionality back on for you, please reach out to us via this support ticket.
Step 3: Sync groups
If you have user provisioning configured at your Organization you might need to resolve potential conflicts with syncing groups to ensure the changes from your Identity Provider during the outage get reflected in your site. To achieve that, please follow the instructions below:
Access https://admin.atlassian.com/
Select your organization
Navigate to Directory > User provisioning. If you don't see User provisioning under the Directory tab, go to Settings and it should be there.
You might see a banner asking you to resolve the syncing conflicts. Banner will show: <group count> groups pending sync(If you don't see a banner, no action is needed.)
Click on ‘Review groups before sync’.
Expand the groups to see what will change (who will be added/removed from that group). You can always add/remove users later at the Identity Provider administration dashboard.This process will link these groups to the ones at your site(s) and allow you to add/remove people based on what we receive from your Identity Provider.
In case it is useful for other customers, here is the timeline of events regarding our restoration:
We are beginning to test the site and verify that everything has returned to its proper place. At first glance, it seems ok, but we are going through it in detail.
Thank you to everyone in this thread for sharing information about this event. It has been very helpful for our organization.
Best of luck to everyone, and thank you all once again.
Thanks for the chuckle. I needed it. Our instance is still down, but hopefully it will be restored before Monday.
Today's support letter below.
Finally we are getting some progress metrics! This is great news.
----
I'm writing to give you an overall incident update. Our team is working 24/7 to progress through site restoration work. At this point, we’ve restored core functionality to 23% of impacted active users and those customers have been notified. Product databases for all other customers are queued up for restoration, which will continue into next week.
We’ve taken a careful and considered approach in the early stages of this restoration process, with the aim of accelerating the restoration process from here.
This incident is our #1 priority. We have mobilized hundreds of engineers who are working around the clock to recover the remaining sites. When your site restoration has started, we will directly notify you via this support ticket.
Sunday Update: a delta of 8% since yesterday.
With this eight-percent-per-day speed and we're currently at approx thirty percent, you can assume the final people being brought back online will have to wait another 8-9 days.
That makes for a neat two-week service interruption. I have NEVER heard of an IT service provider that has a two-week MTTR as a reasonable level of service. Obviously, there are no disaster recovery plans in place at Atlassian. This is VERY worrying.
-----
We want to share the latest update on our progress toward restoring your Atlassian site. Our global engineering teams are continuing to make progress on this incident. At this time, we have restored core functionality to 31% of impacted users. The current stage of restoration is particularly complex due to several steps that are required to validate sites and verify data. These steps require extra time, but they are critical to ensure the integrity of restored sites.
We received an update on our support ticket this morning that said they are ready to begin restoring our site. I've included part of the update below, as it contains information that might help plan their return to service processes.
I've put the last bit in bold, as that's going to be quite important for some organizations:
Once your site is restored, we are looking for your input on a preferred path forward:
- Option 1 (ALL users): In selecting this option, Atlassian will provide access to your site within hours of your selection confirmation. All of your users will be enabled when the site is restored.
- Option 2 (LIMITED users): In selecting this option, Atlassian will work to restore your site to a limited number of users. This will allow you to perform testing before opening access more broadly. If you select this option we will need you to send us a list of email addresses that you wish to have access when we restore the site.
While we have completed validation of Atlassian owned connect apps, please note that third party apps may not be functional. We will work over the coming days to restore these apps. While Atlassian product data will be safe, we can’t provide guarantees about third party app data.
@Shane Doerksen Congratulations! You're lucky. Thank you for sharing the important information to look forward when our site is restored.
Hi @Olimpia Estela Cáceres-Brown . You're quite welcome, and we should all say thank-you to you for starting this thread.
Just as some additional information, we received the update that they were ready to begin restoring our site at 10:36am.
This is what they originally wrote:
"We’ve started the process of site restoration. We are able to move forward with a limited restore of your Atlassian Cloud site. Please note that we don’t yet have an ETA for when this restore will be completed."
They also requested confirmation of which option we preferred (i.e., activate all users or a subset).
I consulted with my colleagues and then replied at 4:26pm that we wanted all users activated.
Atlassian support replied at 9:20pm and said that they would update me again when they had more information.
(all times GMT)
So it has been 16 hours since they first told us that they were starting to restore our site, our site is still down and we don't have an ETA. Our instance is also neither huge nor complex compared to many organizations.
I mention this just as information for other customers, as I'm past the point of complaining about this. Well, almost past the point
If I get more information, I will post it here for the group. Thanks.
So Day 6:
35% of affected customers have now been restored in some way. That is +4% compared to yesterday. Extrapolating that restore speed gives us the last customers being restored in (100% - 35%) / 4% = 16 days time.
I find the following sentences from today's support ticket update rather annoying:
This incident was not the result of a cyberattack and there has been no unauthorized access to your data. As part of scheduled maintenance on selected cloud products, our team ran a script to delete legacy data. This data was from a deprecated service that had been moved into the core datastore of our products. Instead of deleting the legacy data, the script erroneously deleted sites, and all associated products for that site including connected products, users, and third-party applications. We maintain extensive backup and recovery systems, and there has been no data loss for customers that have been restored to date.
Let me ask the Atlassian team: who is responsible for Disaster Recovery and how often do you test the DR process?
It's becoming apparent that they had planned for a complete loss of their data and not for partial loss of data.
This is an architectural problem as well as a DR disaster. Since a subset of their users were impacted, and they probably didn't know it had happened until a period of time had elapsed, they couldn't fall back on (what I presume is their) DR plan of simply restoring everything back to the previous set point. So instead, they are sacrificing us and manually rebuilding our sites one at a time. They are not restoring from backups. They are manually copying data from a restored instance into the prod system. Which is SUPER SCARY.
At least that's my guess based on the information coming out.
We're taking steps in our company to stand up an alternative system so we can meet our release dates. I really feel for the companies who relied on this service for their livelihood.
Just a quick update that it has now been more than 30 hours since they first told us that they were ready to begin restoring our site, and nothing has changed. Our site is still down and we've received no further updates on the ticket.
As @Ulf Sahlin pointed out, their pace of recovery has actually slowed to 4% per day. This is in contrast to what they wrote on April 9th (emphasis mine):
By April 9th they had restored 23% of affected sites.
On April 10th, they restored an additional 8% of affected sites.
On April 11th, they restored an additional 4% of affected sites.
That math says the downtime will be measured in weeks for some customers.
As their other support messages have said, their restoration plans don't immediately include activating third-party apps; they will work on that after the initial restoration. So there will almost certainly be a period of time after restoration where the sites are not yet fully usable for many customers.
Even after all that, Atlassian have said that they can't guarantee third-party data restoration.
Atlassian continually writes that the complexity of this restoration requires verification, etc. Yes, great, we all understand that the Atlassian platform is complex. That doesn't need explained to us.
What needs to be explained to us is:
- why this is a manual procedure at all- why they don't have automated site recovery / disaster recovery tools- how they are verifying our data, and why that's not automated
If their platform architecture is so complex internally that critical management processes like DRP can't be automated, then that is a serious problem.
If they can be automated, then we need an explanation as to why they weren't.
Most importantly, I think we deserve an explanation as to why they don't follow their own recommendations for disaster recovery:
https://confluence.atlassian.com/enterprise/disaster-recovery-guide-for-jira-692782022.html
https://confluence.atlassian.com/enterprise/confluence-data-center-disaster-recovery-790795927.html
Yes, the cloud architecture will be different from data centre or server, but the fundamental principles are very much the same:
1) backup meticulously2) automate everything3) test regularly4) have a runbook so that nobody needs to figure things out on the fly in the middle of a disaster
Point 4 is the most important, as Atlassian's own messages indicate that they are figuring this out as they go.
The service credits that Atlassian offers max out at 50% when their uptime goes below 95% or 36 hours of downtime in a month:
https://www.atlassian.com/legal/sla/service-credits
We're at around 168 hours of downtime, with no end in sight, and we've lost an entire work week to this.
We can't get an actual ETA from anyone at Atlassian and we've exceeded their maximum anticipated downtime by a factor of 5 (so far).
This isn't funny any more.
I suspect that they have all of our sites in a single (!) database and didn't notice that the script had deleted our accounts until the database had significantly deviated from the last snapshot.
Since we reportedly are just 400 sites, or somewhere less than 1% of their customer base, they have elected to sacrifice us to avoid losing data from their other customers by restoring from backup.
So my guess is that they have stood up a snapshot from before the script ran and are in fact manually rebuilding our sites in the prod database. Since this is manually tinkering in the prod database, I suspect they are triple checking their changes to avoid further damage.
To me it speaks of poor architecture and DR planning. They appear have assumed that the only recovery scenario was one where the majority or entirely of the data would be damaged or lost, so they only planned for a complete restoration of the database.
They never planned for a scenario where a portion of the database becomes corrupt (nor did they anticipate that putting 260k plus sites into one management system might be risky.
All of this is hypothetical, but it's the only reason I can come up with for why restoring less than 1% of their system should take 3 weeks and not guarantee that the restored systems will even work correctly.
@Shane Doerksen - no, it isn't fun anymore. It has also been more than 27 hours since they told me they have started the restore process for our site and gave us the same Options (restore all users or a subset of users). No further action. Still down with no ETA. All I can say is ... this is nucking futs!!
@paul.fritz - One would think that if that is the case and they are not doing incremental backups, it may have been much less time-consuming to tell ALL of their customers that they would be down for a brief time-period (4 hours overnight for example) and then take every site down, create another backup that covers the dates in question, restore ALL sites using the full backup and then apply the other backup to ensure sites that weren't affected before, remain unaffected.
But alas, if they are dumping everyone's data together - all I can do is (facepalm), shake my head, roll my eyes and recommend that Atlassian immediately start applying industry standard best practices - I would imagine that this outage will (and understandably should) result in a massive amount of revenue and reputation loss for Atlassian.
I agree. This is why I've been playing the "what if" game. Every scenario I can think of from past experience or industry practices doesn't allow for this apparent very manual restoration.
As for revenue and reputation loss, I wonder what the impact actually will be. With it being such a small number of impacted sites, it isn't making many waves in the technology press. Most customers remain unaffected and oblivious.
Unless somebody goes to one of the larger tech news sites and/or a serious lawsuit gets files, I worry that Atlassian will just chalk this up as a variance and burn what little reputation they have left with us. Being kinda pessimistic as just got off of Zoom with my leadership asking that we look at alternative systems....
Just another quick update that our site is still not back.
It has been 51 hours since we first received an update to our support ticket that said they were beginning restoration.
Nothing has changed and we do not have any kind of ETA.
According to https://opsgenie.status.atlassian.com/ they have restored another 5% of users since yesterday, so it looks like that 4% to 5% per day rate is probably what they're going to maintain. Unless they accelerate that significantly, some people are likely looking at almost 2 more weeks of downtime.
Hello,We apologize for the length of this ongoing incident and for not being more proactive in our communication with you. We understand how mission-critical our products are for you and want to make sure we are relaying the most accurate information possible.
While we really want to get in front of you live to answer your questions, we are prioritizing getting customers up and running first and foremost. We will host an AMA (Ask Me Anything) after we get all of our customers fully restored.
In the meantime, please add your questions here and we will respond as quickly and transparently as we can. Some questions may not be answered until we do an official PIR, but we will let you know that and answer as much as we can now.
Our Chief Technology Officer Sri Viswanath has posted a blog about this incident with more background and details, and our team is working around the clock to move through the various stages for restoration.
We are working 24/7 to restore your service. Thank you to those who have been sharing all the information you know from your support requests. Thank you for being open, honest, and caring for your fellow Community members. This speaks a lot about what makes Community a special place.
@Olimpia Estela Cáceres-Brown Thank you for raising this post.
@Shane Doerksen Thank you for being open and honest with your feedback and updates.
@Karla Thank you for the feedback and updates.
@Ulf Sahlin We hear you and appreciate your feedback and updates.
@paul.fritz Thank you for the honest, direct, and needed feedback.
@Karim Abrik We've heard your feedback and appreciate it.
Lastly, if you're unsure if a support request was raised on your behalf, please let me know, and I will personally check. My team and I went through all Community posts related to this incident to ensure everyone's site had a support request.
Regards,Stephen Sifers | Product Lead, Community
So my group just had our conversation with Trisha and Diane from Atlassian, and here are my takeaways.
Once we are notified that our data is being restored, all of the components will be restored at the same time (Confluence, Service Management, and Insight in our case). Once notification occurs it will take roughly 4 to 5 days before we are able to access our site.
Due to how the data was deleted, the site and URL was also deleted. Site and URL will be recreated/restored and we were told that we are on there radar even though our URL is no longer visible.
Atlassian is restoring in batches of 60 customer sites, but they are unable to tell us what batch we are a part of or when to anticipate the initial email stating that recovery of our site is beginning.
The restoration is a manual process and they are re-adding the deleted sites into the data stores of the unaffected sites. Asked about the possibility of restoring all 400 effected sites to their own data store to get users up quicker did not return any answers.
So for now, we will wait until our ticket number is called (email stating data restoration has begun) and then able to head up to the counter to wait the 4 to 5 days until we can begin to access/verify our data.
Congrats on your site restoration. Thank you for providing the timelines as well. We have not heard anything on our site entering the restoration phase, so I'm guessing we are still at least 5 days away from possibly having our data/site restored, but very much appreciate you documenting and sharing your journey.
And as you so elegantly stated:
"May your instance always be online and your data always be backed-up."
Our site was restored this morning and we also received the same message. Very similar timeline as mentioned by @Shane Doerksen
Thanks for continually posting the updates!
Excellent @Shane Doerksen - We're inLine with you, we're almost ready to open to all MIT Libraries staff. This posting really made a difference!
Happy Easter! Happy Passover! Happy Ramadan!
Our site went online this morning, all data items seem to be present. Yay!
That's great news @Ulf Sahlin ! Glad to hear it.
Our site came back online at the end of Friday, but the stresses of doing our release without Jira kept our team from reviewing it until today. Everything seems restored except for our Insights data. We think it may be missing some assets. We are working with Atlassian to check.
Ironically the system that we had to migrate to from the system that had worked okay for us.....
Glad to hear that your site is back @paul.fritz . We are still reviewing our data here. Most of it seems to be okay, but we may have some small issues with the ownership of our service request tickets. But at least the site is up, so this week is off to a better start than last.
@paul.fritz
Very happy to hear y'all are able to access your products again! I see you updated your support case with the missing assets. I'll personally update the team as well to ensure they're aware of your issue post restore.
Regards,Stephen Sifers
It looks like you're new here. Sign in or register to get started.
