nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

REST API returns some duplicate pages for content?page queries

Daniel_Miller

Via Python and the "requests" package, using the REST API, I've written scripts to read and write the contents and add attachments to Confluence pages in a particular workspace. Before each transaction, my code verifies that a page of the proper title exists within the workspace by retrieving all the page titles, using the syntax:

my_url/rest/api/space/<spacekey>/content?type=page&limit=25&start=0&expand=space,version

I call this iteratively, incrementing the start by 25 until the number of returned items is less than 25. I look for the desired page title within the set of titles returned. A total of 276 page entries are returned.

For a few new pages, the above logic is failing to find the page title within the space. HOWEVER, the page is visible within the Confluence GUI, apparently in the right place, and if my script ignores the error it can indeed read and write contents and add attachments to the page!

I investigated by comparing the list vs. the set of all page titles returned by the above logic. What I found was a number of duplicate page titles, and the number (about 60 or 70) and page titles of the duplicates varied each time I ran the script. It seems as if Confluence is getting confused about the indexing of pages when I'm retrieving them by start number... or I've made some weird error.

Has anyone seen a similar issue?

Accepted answers

All comments

Stephen Sifers

Hello Daniel,

The result you are reporting can be due to the amount of pages with versions. You will see multiple page returns with duplicate names but note the versions and ID will be different for each duplicate.

Another way to verify the API is not returning duplicate results would be to run a manual comparison.

Simply take your API URL and paste it into a browser, the return should be the JSON results

Save each result page or paste results into a text editor

As you’re doing in your script, keep incrementing the start till you reach the end.
Once you’ve worked through all of the results, use a JSON parsing tool to compare the results
1. You may have to move your results out of the parsing tool to record all of the results in one place.
If you’re seeing identical duplicate results within your comparison, then you may have a Confluence indexing issue.

If the Confluence index is the issue, you can rebuild the index to resolve this issue. Here are additional resources to help with resolving index issues:

Content Index Administration

How to Rebuild the Content Indexes From Scratch on Confluence Server

Please let us know if you’re still having identical duplicate results within your API call after comparing your results and evaluating if an Index issue is in fact present. If your issue persists, we will most likely need review further to see if there is a defect within the Confluence version you’re on.

Regards,
Stephen Sifers

Daniel_Miller

Stephen: I don't believe we have pages with duplicate names. Note that the original problem was that this routine did not return a page I know to exist. In any case, when repeating the test, I get varying sets of pages returned more than once. What I'm seeing is a varying set of false duplicates "crowding out" other, actual pages.

When looking at the results, I often get a different first page. How does Confluence "order" the pages returned in this query? Should there be a static order (assuming no pages are added or removed from the space)? If not, how does Confluence guarantee that when you request (for example) pages 100-124, you don't get a page returned previously? This seems to be what is happening.

Also, as a workaround, I tried increasing the number of pages per batch to 200 (the apparent maximum). It takes two queries (0-199 and 200 onwards) to retrieve all the pages and the second query returns 76 pages with NO duplications of the first 200. So I have a way forwards, though I'm still concerned there is an issue here which could cause downstream problems as more pages are added.

Thanks

Stephen Sifers

Hello again Daniel,

When I tested on a fresh install of COnfluence 6.13 I was getting non-random results when using a GET method within a space.

With this said, you may want to explore using the SEARCH call which will allow you to apply CQL which then allows you to set Order By on your results. This should produce a more predictable result in your case. You may find out further information on REST API Search Content along with Advanced Searching using CQL.

Any additional information you have for your Confluence deploy would be greatly appreciated so we may test and ensure there is not a defect in the product. Please let us know the following:

Confluence Version
Any add-ons used within Confluence
Database application
Database Collation

This should allow us to test further and attempt to reproduce the issue.

Regards,
Stephen Sifers

Daniel_Miller

Stephen: I assume you were using a small batch size compared with the overall number of pages in the space - e.g. 25 vs. 275?

Here is the requested info:

Confluence Version 6.6.7.

The database is postgres 9.6.

No special rules for collation.

Add-ons used within Confluence:
· Bob Swift Atlassian Add-ons - Cache
· Comala Workflows
· Multiexcerpt Plugin4
· Page Info for ScriptRunner
· Scroll Word Exporter
· Staffing Timeline
· Adaptavist ScriptRunner for Confluence
· Angular JS integration for AUI (AUI-NG)
· Bob Swift Atlassian Add-ons - Advanced Tables
· Bob Swift Atlassian Add-ons - Table Library
· Confluence Source Editor
· Forms for Confluence
· Keysight Admin Tools for Confluence
· Scroll Exporter Extensions
· Scroll Runtime for Confluence
· Spreadsheets

Nejc Grenc

Hello,

In our organization, we recently encountered a peculiar issue with Confluence content retrieval. After migrating to Atlassian Confluence 7.19.8, we encountered a discrepancy in the generated reports, specifically the absence of certain pages. In my investigation, I traced the issue to the REST calls responsible for retrieving pages from Confluence. The problematic behavior exhibited similarities to those described by Daniel Miller.

Problem description

Previously, we employed consecutive calls, each limited to fetching 100 pages, to retrieve Confluence pages. For the specific space in question, which contained 255 pages, we executed three calls using the following format:
{{confluence-url}}/rest/api/space/{{space-key}}/content (?start=0&limit=100, ?start=100&limit=100, ?start=200&limit=100).

The initial call yielded 100 distinct pages.
The second call provided 100 pages, of which 25 were exact duplicates of those obtained in the first call. These duplicates encompassed all page attributes (e.g., ID, title, and status).
Subsequently, the third call returned 55 pages, out of which 19 were duplicates of those retrieved in the previous two calls.

In total, the correct number of pages (255) was obtained, but there existed 44 duplicates, signifying the omission of the remaining 44 pages within the space.

This peculiar behavior persisted across standard REST calls (executed via browsers or Postman), as well as when utilizing Java's SpaceService component.

The returned pages exhibited no discernible order, although they consistently followed the same sequence. In other words, each call consistently returned the same set of pages in the same order, as long as no modifications were made within Confluence (i.e., no pages created/updated/deleted). This pattern also encompassed the presence and positioning of duplicates.

The solution ?

I implemented the Daniel Miller's proposed solution, involving the utilization of the maximum permissible limit value, which is 200. It seems this approach is effective, as I encountered no further duplicates in my tests.

Granted, my sample size is small (and I have no time to investigate this much further), but there appears to be a deeper issue in Confluence's content retrieval that warrants further investigation.

Regards,
Nejc

Daniel Miller_45nljd

Nejc: It has been a few years, but my recollection is that a larger limit size turned out to reduce the number of duplicates, but not eliminate them completely. We never pursued this issue because, as noted, the "missing" pages were actually there and could be read from and written to by the scripts.

Nejc Grenc

Hello, Daniel!

I understand your message. The Confluence pages are currently fully accessible within the Confluence platform itself through browser interactions or direct REST calls. However, these pages were inadvertently omitted from our reports generated using the mentioned REST content call.

Oh, well. Currently, this solution appears to be stable; however, it is likely that we will need to address this issue again in the future if duplicate pages continue to cause them to go missing.

I sincerely appreciate your contribution in providing the initial solution and reminding us of the potential for future failures.

Thank you, Daniel!

Heinrich Ulbricht (WikiTraccs)

Wow, this is just on time! I recently got a support ticket for the WikiTraccs Confluence to SharePoint migration tool that I'm building that reported errors in migration progress reporting. After digging I found that Confluence returned duplicate pages as well, about 10% in a space of about 4000 pages. Page size was 100 in this case. The solution for now: ignore the duplicates.

After reading through the posts I'm just a bit worried that pages might also be missing. Did anybody try to rebuild the content index (as suggested by @Stephen Sifers) to solve this problem, and did it help?

I'll probably increase the page size to 200 as well. Is this a hard limit or could that be changed somewhere in the environment?

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

server

confluence-server

rest

rest-api

atl-adm