[Resolved] Incident - Data Lag in Atlassian Data Lake Update

Hello Community,

First, we want to acknowledge that your data in the Atlassian Data Lake is of utmost importance and that data integrity is crucial in driving your business needs using Atlassian Analytics.

We want to share an update on the ongoing data lag incident in the spirit of transparency.

πŸ”” Incident

In the past two weeks, we were made aware of data inconsistencies in the Atlassian Data Lake. We started investigating and restoring data in the lake. While we have completed an initial restoration, other issues were identified that have required additional fixes.

We are continuing to monitor the situation and appreciate your patience. Please know that it is the top priority for our team to work to resolve this incident.

⚠️ Data inconsistency remaining 

Some data inconsistencies on the jira_issue and jira_issue_history tables are still actively being fixed. We have completed a data refresh across all regions and are now verifying the resolution of the inconsistencies in the jira_issue and jira_issue_history tables.

These discrepancies are affecting customers with data residency enabled in Europe and the US, and also customers without data residency enabled.

πŸ” Root cause

We have identified the root cause and plan to take action to ensure this does not happen again.

The problem stemmed from an edge-case scenario on how we process files due to changes in the Data Lake systems. We rolled back those changes to ensure files are processed correctly to restore the data as soon as possible.

⏭️ Next steps

We are now verifying that the remaining inconsistencies are fixed and will provide our next update once the verification is complete.

A PIR (Post-mortem Incident Review) will be conducted to ensure that we take the appropriate actions to avoid this happening again in the future.

 

We are very sorry for the disruption caused to your teams. Please rest assured we are doing everything we can to restore the service and protect data quality.

Claire and the Atlassian Data Lake team

1 comment

Claire Gayrard
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 31, 2023

Hello Community,

We are pleased to report that all the missing or inaccurate data has been restored and verified.

We are monitoring the data in the Atlassian Data Lake.

❓See something wrong

If you come across any anomalies in your data, kindly create a support ticket and we will investigate the issue.

⏭️ Next steps

A Post-mortem Incident Review (PIR) will be conducted to ensure that we take appropriate actions to prevent this from happening again in the future.

 

Again, we are deeply sorry for the disruption caused to your teams.

Claire and the Atlassian Data Lake team

Like β€’ # people like this

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events