nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How Should I get all text from all spaces and their child content?

Humashankar VJ

Hello - Looking for some insight on this problem statement:

How to retrieve all content from all global spaces in your Confluence instance

Problem Statement:

I want to get all the content from all spaces with type "global".

This now costs more than 70 API calls. This is because there is a limit of 25 objects per call and I need to get the "space key" from every space manually to get their content via a for loop with calls to "https://wiki-acc.[mycompany].com/rest/api/space/[space_key]/content?expand=body.storage".

If there is a way to limit the amount of API calls to achieve my goal, I would very much like to know. It now takes about 37 seconds to execute, which is too slow for my use case.

Thanks in advance!

Reference ticker and latest code:

How can I get all text from all spaces and their c... (atlassian.com)

CC: @Bas Hulskamp

Warm Regards

Accepted answers

Humashankar VJ

Hi @Barbara Szczesniak

Can you take a look in this use case and help @Bas Hulskamp

Regards

Bas Hulskamp

Hi @Humashankar VJ @Aron Gombas _Midori_ @Barbara Szczesniak

Thank you all for your assistance!

I have solved my issue now, by using async functions to retrieve the data with the Confluence API. This took the execution time down to <=3 seconds from the +/-60 seconds that it was before.

This is my code

async def get_all_global_space_keys(self) -> list[str]:

        url = self.base_url + "/rest/api/space"

        params = {"type": "global"}

        all_keys = []

        async with aiohttp.ClientSession() as session:

            while True:

                async with session.get(url, headers=self.headers, params=params) as response:

                    data = await response.json()

                    for result in data["results"]:

                        all_keys.append(result["key"])

                    next_url = data['_links'].get('next', None)

                    if next_url:

                        url = self.base_url + next_url

                    else:

                        break

        return all_keys

    

    async def get_content_from_space(self, session, space_key) -> list[Document]:

        documents = []

        url = self.base_url + "/rest/api/space/" + space_key + "/content"

        params = {"expand": "body.storage"}

        async with session.get(url, headers=self.headers, params=params) as response:

            data = await response.json()

            for result in data["page"]["results"]:

                content_url = self.base_url + result["_links"]["webui"]

                # content: str = BeautifulSoup(result["body"]["storage"]["value"], 'html.parser').get_text()

                content: str = result["body"]["storage"]["value"]

                documents.append(Document(content, metadata={"source": content_url}).to_json())

        return documents

    async def get_all_content_from_all_global_spaces(self, space_keys: list[str]):

        documents: list[Document] = []

        async with aiohttp.ClientSession() as session:

            tasks = [self.get_content_from_space(session, key) for key in space_keys]

            results = await asyncio.gather(*tasks)

            for result in results:

                documents.extend(result)

        with open(self.WIKI_DATA_PATH, 'w', encoding='utf-8') as file:

            file.write(json.dumps(documents))

    async def get_all_async(self):

        space_keys = await self.get_all_global_space_keys()

        await self.get_all_content_from_all_global_spaces(space_keys)

c = ConfluenceService()
asyncio.run(c.get_all_async())

All comments

Bas Hulskamp

Anyone?

Bas Hulskamp

@Humashankar VJ I'm not familiar with the people who are active on this forum. Can you perhaps @ mention someone that would know more about my question?

Humashankar VJ

Sure @Bas Hulskamp - Let me work on that part.

Humashankar VJ

Hi @Barbara Szczesniak

Can you take a look in this use case and help @Bas Hulskamp

Regards

Barbara Szczesniak

The reference question is tagged with Data Center, but this question is tagged for Cloud.

I work with Cloud, and I do not use Rest API, so I'm not sure I can provide assistance.

My first questions are:

What are you using the content for—what is the reason you need all of the content? This may impact the format you the content in.
Why not export the spaces to the format you need?

Humashankar VJ

Hi @Bas Hulskamp - Kindly take the questions for Barbara

Bas Hulskamp

Good afternoon @Barbara Szczesniak ,

I use the REST API tot query an on premise instance of Confluence.

All the text of pages and files I use as content for my RAG (Retrieval Augmented Generation) application. RAG is an AI principle if you didn't know.

It now takes 74 seconds to get all content of all spaces (as explained in the linked post), because I have to call the API at least an amount of times equal to the amount of spaces to get all the content with this endpoint "https://wiki-acc.[mycompany].com/rest/api/space/[space_key]/content?expand=body.storage". Not to mention having to iterate over the spaces to get the SpaceKeys, with a limit of 25 per call because I couldn't get the total amount of spaces from an endpoint (see code in original post).

I would like to know if there is an alternative way that takes less API calls and is faster.

I'm not sure what the "export spaces" feature is that you mentioned... how does that work? Would that help me in this case?

Aron Gombas _Midori_

There are at least two opportunities for parallelization:

When you get the first page of the spaces, you can immediately start collecting the contents in those spaces. In other words, you don't need to load all spaces before you can start collecting the pages in those.
You can, obviously, collect the pages in two separate spaces in parallel.

Using these two trivial tricks, you can massively accelerate the data collection here.

Note that you may run into rate limits if you go over-aggressive with parallelization. So it is a bit of experimentation and tuning.

Humashankar VJ

Thanks @Aron Gombas _Midori_ for the insight

Hi @Bas Hulskamp - Can you intake the above data points and see how you want to move on ?

Bas Hulskamp

Hi @Humashankar VJ

I'll see what I can do with this. Thanks for your insight @Aron Gombas _Midori_ !
Seems a bit like an AI answer by the way haha

Bas Hulskamp

async def get_all_global_space_keys(self) -> list[str]:

        url = self.base_url + "/rest/api/space"

        params = {"type": "global"}

        all_keys = []

        async with aiohttp.ClientSession() as session:

            while True:

                async with session.get(url, headers=self.headers, params=params) as response:

                    data = await response.json()

                    for result in data["results"]:

                        all_keys.append(result["key"])

                    next_url = data['_links'].get('next', None)

                    if next_url:

                        url = self.base_url + next_url

                    else:

                        break

        return all_keys

    

    async def get_content_from_space(self, session, space_key) -> list[Document]:

        documents = []

        url = self.base_url + "/rest/api/space/" + space_key + "/content"

        params = {"expand": "body.storage"}

        async with session.get(url, headers=self.headers, params=params) as response:

            data = await response.json()

            for result in data["page"]["results"]:

                content_url = self.base_url + result["_links"]["webui"]

                # content: str = BeautifulSoup(result["body"]["storage"]["value"], 'html.parser').get_text()

                content: str = result["body"]["storage"]["value"]

                documents.append(Document(content, metadata={"source": content_url}).to_json())

        return documents

    async def get_all_content_from_all_global_spaces(self, space_keys: list[str]):

        documents: list[Document] = []

        async with aiohttp.ClientSession() as session:

            tasks = [self.get_content_from_space(session, key) for key in space_keys]

            results = await asyncio.gather(*tasks)

            for result in results:

                documents.extend(result)

        with open(self.WIKI_DATA_PATH, 'w', encoding='utf-8') as file:

            file.write(json.dumps(documents))

    async def get_all_async(self):

        space_keys = await self.get_all_global_space_keys()

        await self.get_all_content_from_all_global_spaces(space_keys)

c = ConfluenceService()
asyncio.run(c.get_all_async())

Humashankar VJ

Great to know @Bas Hulskamp - Have a great one, let me mark this as solved for other reference in the future

Aron Gombas _Midori_

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

cloud

confluence-cloud

data-center