nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Delay is response for Confluence Cloud

Pranab Bhatta

Hi All,

While working on Confluence Cloud API (Search content by CQL), there is a property start (The starting index of the returned content), where we can give the integer value. We are trying to fetch all the pages using the "Search content by CQL". As the we keep on increasing the start value, i.e. greater or equals to 10000, the response time increases to around 30 seconds per request. Otherwise the response time is less than 5 seconds. This is hampering the the overall performance.

Not sure if this is a bug. Could you please guide us?

Thanks,

Pranab

Accepted answers

All comments

Nic Brough -Adaptavist-

Returning 10,000 pages is a lot of work, it's going to take time.

I'd suggest not doing that, and using the system as it is intended to be used instead (i.e. why are you trying to get so many pages? That volume is useless to an end user, so you must be trying to do something other than provide pages to users which is either pointless, or for some reason where other reporting will be a far better method)

Pranab Bhatta

Hi Nic,

Thanks for the reply. I am not trying get 10,000 pages at a single request. If you check the API, there is a start parameter (The starting index of the returned content). We are getting the pages in batches. Initially start value is set to 0, then I will increase the start by 50 with each batches. When the start index reaches the 10,000, then I notice the considerable delay.

Here is the documentation link https://developer.atlassian.com/cloud/confluence/rest/#api-content-search-get

Thanks,

Pranab

Nic Brough -Adaptavist-

You're still asking it to work with a large number of results and it needs to build, hold and work through a huge list.

I'd suggest you look at simplifying what you're doing to a sane range of pages.

Pranab Bhatta

Hi Nic,

Thanks for the reply.

Sorry for asking the question again. Let me explain again. Consider you have over 1,00,000 pages in Confluence and you would like to retrieve all pages.

1) You are making 1st API call, something like this /wiki/rest/api/content/search?cql=type=page&start=0&limit=20]

2) Confluence send responses something like below with 20 records. This is not a large number of results set.

"_links": {
"base": "https://ibmappcon.atlassian.net/wiki",
"context": "/wiki",
"next": "/rest/api/content/search?limit=20&start=20&cql=type=page",
"self": "https://ibmappcon.atlassian.net/wiki/rest/api/content/search?cql=type=page"
}

3) Again you will make an API call by extracting the value from the next property above in the response. You will perform this call till all the 1,00,000 pages are retrieved.

4) So when you reach at 10000 index. Your API call with the next value will be like this [/rest/api/content/search?limit=20&start=10000&cql=type=page]. You will see considerable delay in the response.

So my question, in the above scenarios though my per page size is 20 records, there is delay in the response.

1) Is there a better way of fetching the all the pages with pagination?

2) Will it improve the performance, i.e. response time?

Thanks,

Pranab

Lawrence Chou

We are also trying to crawl pages with search API like Pranab does. We didn't use other RESTful endpoints for crawling due their respective bugs and limitation.

One workaround we are currently trying is preventing big "page" with sliding window, like:

1. CQL="type=blogpost and created > T1 and create <= T2", page=1

2. CQL="type=blogpost and created > T1 and create <= T2", page=2

3. CQL="type=blogpost and created > T1 and create <= T2", page=3

4. CQL="type=blogpost and created > T2 and create <= T3", page=1

...

10000. CQL="type=blogpost and created > T1001 and create <= T1002", page 3 <---- small "page"

instead of:

1. CQL="type=blogpost", page=1

2. CQL="type=blogpost", page=2

...

10000. CQL="type=blogpost", page=10000 <---- big "page"

P.s. I can imagine Nic's point of "hold and work through a huge list" may be the cause of such limitation, but I still think it's a bug due to underlying implementation. Above is just a possible workaround to this bug.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

cloud

confluence-cloud