Confluence Server and Data Center: Content duplication fix and next steps

[#1]

Hi everyone,

My name is James, and I’m an Enterprise Support Engineer on the Confluence Server and Data Center team. I’m here to share some updates and information with you regarding some issues we’ve been facing in Confluence.

Recently we’ve seen a spike in user reports of content duplication in both Confluence Server and Data Center products. Firstly I’d like to say that we have been closely following the comments and reports from users in the various tickets I’ll detail below. Over time there has been growing frustration at the perceived lack of progress and updates from Atlassian, and so we wanted to ensure we are open about the challenges, and in providing updates and information about the problems impacting you and your users. Moving forward, we’ll use this blog post as a centralised “one stop” location that we’ll update as our understanding of the problems evolve and grow.

Content duplication issues under investigation

Content duplication is when either an entire page, or a subset of content on a page unexpectedly appears twice in the Confluence editor whilst collaborative editing is enabled. This duplication is most apparent when a page is published and the content appears longer than the user expected. Content duplication can also occur more subtly with only small subsections of content duplicated. This unintended duplication is frustrating to users as it requires manual intervention to correct, or perhaps worse, may not be noticed at all.

Content duplication is a symptom of several underlying problems with collaborative editing. When the collaborative editor encounters a scenario it is unsure of how to handle, it will keep both versions of the content in order to allow the user to recover the content they want to keep. There are multiple scenarios where we provide the collaborative editor with conflicting information, and it falls back to this safety mechanism, to preserve all the content.

The challenge has been that each of these issues has been incredibly complex to investigate due to limited visibility of data involved in the issue, and that the behaviour of each of the issues appears identically to an end user. We understand this may have given our users the impression that fixes have been slow and still haven’t worked, and that Atlassian isn’t treating the issue as a priority. Behind the scenes, the various tickets are being regularly reviewed and work is continuing on fixing the various causes of the behaviour being experienced by users. Throughout the process of investigating and fixing these issues, we’ve received dozens of Collaborative Editing Feedback Tool reports (more on this below) to review. We really appreciate the assistance from the customers who have contributed to this investigation as it has led to the identification of the issues below.

Issue	Cause summary	Status
CONFSERVER-59227 - Duplicate content shown when editing a page CLOSED	A series of faults can occur during editing that results in Confluence comparing the existing page with an empty page. This causes Confluence to treat all the content on the page as new content from another editor resulting in content duplication of the entire page.	FIX RELEASED - Confluence 7.12.3 LTS Backports Confluence 7.4.14 - FIX RELEASED Confluence 7.13.0 - FIX RELEASED
CONFSERVER-69074 - Duplicate Content Shown When Editing A Page After Modifying An Attachment CLOSED	Modifying a page attachment, or a related page with an attachment sometimes resets the editor state, resulting in Confluence duplicating the entire page during editing.	FIX RELEASED - Confluence 7.15.0 LTS Backports Confluence 7.4.15 - FIX RELEASED Confluence 7.13.4 - FIX RELEASED
CONFSERVER-73383 - Duplicate Content Shown When Editing A Newly Published Page CLOSED	Confluence provides an unexpected page version to the collaborative editor resulting in the use an incorrect editor state when merging. This causes Confluence to duplicate the entire page content.	DUPLICATE  - See below for details.   Note: We previously believed that the fix for CONFSERVER-69074 would also fix this issue, but unfortunately we’re no longer confident this is the case.  We have since confirmed that this issue is resolved by the fix for CONFSERVER-69074. We additionally confirmed that some plugins can transparently trigger this issue during page creation and publishing.
CONFSERVER-59993 - Duplicated content in pages after running Synchrony Hard Data Removal LONG TERM BACKLOG	Using the Synchrony Hard Data Removal task in Confluence can cause duplication on some pages. This issue requires manually running a task and cannot occur without Administrator intervention.	Closed   Note: We were unable to reproduce this issue, and believe that it may be an occurrence of the other bugs given the feedback from the investigating engineer. Please check the issue ticket for additional information.

Thanks to subsequent Collaborative Editing Feedback Tool artefacts, we can see that the fixes we’ve implemented have worked, but there are still more issues to resolve. As we continue to correct these issues, it is possible we will identify new issues that are currently being masked by those we’re aware of. As such, additional data provided by all of our customers is going to be critical in resolving any issues we’re not yet aware of. If you’re able to help us investigate, please see the How you can help section below for more details.

We know this collection of bugs is high impact for you and your users, so they will remain high priority for our team until we’re comfortable we have the issues resolved. We’ll continue to communicate with you throughout this process.

Understanding The Fixes

With backports for the most recent issue complete, I want to explain the impact these fixes will have. A version of Confluence that has the above fixes should not create new duplicated content. However, content that was already duplicated can still remain, due to how Confluence handles publishing of pages and the creation of drafts.

When a page is published, Confluence creates a new draft of the page in the background for easy access via the REST API, database etc. As some of the bugs cause content duplication in the page draft at the time of publication, this can lead to a situation where a page shows the expected content at the time of publishing, but in the background, the draft contains duplicated content. Next time someone edits the page, it appears as if the content duplication has just occurred, whereas the duplication actually occurred at the time of publishing, not editing. We call this stale duplication, and it may be present in your site after upgrading. 

We’ve addressed all of the causes of duplication that we’ve been able to identify so far, and will continue to monitor for fresh content duplication reports from the community in case there are additional causes that we’ve not yet identified. Unfortunately, we have not been able to devise a way to determine whether duplication is stale, or a new case. For this reason, we ask that you only submit duplicate content reports when the duplication has occurred on pages created (as opposed to published) after you upgraded to the fixed version. It would be great if you could follow the method detailed in the “How You Can Help” and “How to Open A Support Ticket” sections.

We expect this simple rule will help us to rapidly identify new occurrences (should there by any), and begin remediating immediately.

Additional improvements to collaborative editing underway

Getting a fix out to you all remains our primary focus, but there are some other things we’ll be doing in parallel.

We have reviewed the bug descriptions to make them clearer and easier to understand. The continued community conversation on these bugs after releasing fixes shows we’ve not done a great job at this.

As the issue has grown yet more complex, we’re going to formalise the support pathways for investigating these issues. We will be shipping the Collaborative Editing Feedback Tool we have been using to collect reports from customers as part of Confluence in a future release. This will give administrators easy access to generate support artefacts that we can use to investigate future issues. We’re updating our documentation on this tool now and will let you know when it’s in the product.

To enable us to improve the health of the collaborative editing ecosystem as a whole, we’re looking to add more detailed analytics into the system, and to leverage these analytics to improve fault detections and visibility moving forward.

How you can help

So far we’ve received and analysed dozens of artefacts from the Collaborative Editing Feedback Tool which have helped us identify the issues detailed above. While we can’t thank the users who have contributed to this process enough, we could definitely use more help.

We have a knowledge base article that includes the details of how to use the tool, and what information is collected. Please review this page carefully.

How to use the Collaborative Editing Feedback Tool to provide problem reports | Confluence | Atlassian Documentation

The article includes a link to download the plugin.

How to open a support ticket

To prioritise the investigation, and ensure the files are routed to the team dedicated to this issue, you can open a support ticket using the following guide

1. Open a new ticket at https://support.atlassian.com with the following information

   • What can we help you with? Technical issues and bugs

   • Which product is this for? Confluence

   • Where is your Confluence hosted? Self or 3rd-party hosted (Server or Data Center product)

   • What is your Confluence Support Entitlement Number (SEN): Please enter your SEN here

   • Summarize your issue: Duplicate Content Data Report

   • What is the impact to your business? Level 4

   • Give us more details: Please paste the below snippet

     Hi Support,

     I've collected some data and opened this ticket to help investigate the duplicate content issues as detailed in this community post: Confluence Server and Data Center: Content duplication fix and next steps - Atlassian Community 
     Please find the Collaborative Editing Feedback Tool artefacts attached.

     Ref: e0c45be755
     
     Thank you!

   • Which version of Confluence are you running? Please select your Confluence version

   • Fill out the remaining fields. These fields will vary depending on your support entitlement

2. Once the ticket is created, navigate to the provided ticket ID (or via My Open Tickets)

3. Create a Support Zip

4. Attach the reports and the support zip to the ticket

These investigations are incredibly complex and time consuming with each artefact taking in excess of half a day to process and investigate. With this in mind, there will be a delay in responding to tickets with results and updates. We value all of the data provided to us, but we want to set expectations based on our experience to date.

I’ll continue to update this post as we have new information to share. Thank you all for your patience thus far.

[Bill Taroli]

So, question... because it's unclear if this page has been updated, per notes in CONFSERVER-69074 

Right now the situation (as posted) seems to be that known issues are patched (but 7.13.4 and 7.4.15 are yet to be released) but that there is the long-term hard data removal issue, that itself seems very low risk.

So are there NEW issues not yet reflected here? We are hearing from our IT team that they are told Synchrony is dangerous even as recently as 1/11. It would be good to demystify some of this so we understand what issues are really out there and when it might be safe to proceed.

[Tim Hailey]

@James Ponting Are these fixes included in all versions beyond 7.13.0 or only the following:
https://jira.atlassian.com/browse/CONFSERVER-59227    Confluence 7.13.0 only?
https://jira.atlassian.com/browse/CONFSERVER-69074    Confluence 7.13.4 and 7.15.0 only?

Thanks,

Tim Hailey
Atlassian Systems Administrator

[James Ponting]

Hi All,

I just wanted to leave a quick note on this community post in the same vein as the update on the various tickets.

At this point we've not seen any additional reports indicating any new examples of duplicated content. With the reduction in reports of issues and not having seen any new duplication, we believe that this issue has now been resolved.

With that in mind, we're going to discontinue monitoring this post and the associated tickets. If you are able to produce new examples of content duplication inline with the expectations set out above, please contact our support team at https://support.atlassian.com and they'll be able to assist you further.

We've undertaken the work discussed in the "Additional Improvements" section above. There are some additional changes that are required that make it likely that this work will land after the next LTS release, however we have the ability to release this work as part of the JAR file used in the diagnostic steps, and will do so if required.

Finally, I just want to thank everyone for their understanding and patience during this painful experience, and for the numerous reports submitted by you all. It would have taken a lot longer to find without all your help.

Thanks,
James Ponting
Engineering Manager - Confluence Data Center 

[Ollie Guan]

This is not serious and the user can fix it manually.

But once this happens, it's really a headache!