Searching fields Confluence database AND text within the Confluence page

[David Stokes]

I am deep-diving into how to make content in Confluence more searchable, ie: text contained in the page AND metadata associated with that page.  I've gone through the different options:

• Confluence - metadata in a page properties macro containing a table with each of the metadata values
  • Can't search field by field
• Confluence Database
  • I kept coming up with things that don’t work as I want, so I have been looking at other options – still lots of room for improvement for Confluence Databases
• Confluence Database with fields from a Jira record
  • Each page has a Jira record that contains its metadata (which includes using a Jira field with a select list with a set of preexisting tags that can be controlled, as opposed to page labels, which aren't controlled), meaning that I can use the power of JQL to search the metadata
  • But how can I search both Jira / Confluence database AND the content on the linked Confluence page?

[Tomislav Tobijas]

Hi @David Stokes ,

I would say that this is definitely a limitation (in a way) when it comes to searching across products/apps.

In short:

• Page Properties Macro - you can store metadata within it, but Confluence's search doesn't allow you to search by individual fields.
• Databases - there's this feature request that might make some lives easier: CONFCLOUD-79086: As a user I would like to import Confluence pages into Confluence Databases using CQL queries 
• Jira integration - from what I know, there's no built-in way to simultaneously search the content of the linked Confluence page (e.g., from Jira/JQL)

Not sure if Rovo could be more helpful here? For example, something like this: Knowledge sources for Rovo agent scenarios 

You could also check/build something like this: How to find issues linked to a specific Confluence page using JQL?, but I'd say it's definitely not ideal 🫤

Cheers,
Tobi

[David Stokes]

@Tomislav Tobijas  thanks for your thoughts

Here's a hack that I think might work:

Take a copy of the raw text in a Confluence page and paste it into the Description field on the Jira record, so that it can be used in the JQL query.  I would have an automation so that when the page gets updated, it updates the Description field on the Jira record. But there are limits on this approach for pages above a certain number of characters (I think it’s 32,000 characters).

What do you think?

I just need to figure out the automation.

[David Stokes]

OK, so I've got it to give me the contents of the page, but it's in wiki markup / HTML:

{id=446627872, type=page, ari=ari:cloud:confluence:e32f8278-1d23-4842-bd31-e2f6853dc3b3:page/446627872, base64EncodedAri=YXJpOmNsb3VkOmNvbmZsdWVuY2U6ZTMyZjgyNzgtMWQyMy00ODQyLWJkMzEtZTJmNjg1M2RjM2IzOmNvbnRlbnQvNDQ2NjI3ODcy, status=current, title=Test Document (DOC12), macroRenderedOutput={}, body={storage={value=<p local-id="1db4738ba965">Here is some text</p>, representation=storage, embeddedContent=[], _expandable={content=/rest/api/content/446627872}}, _expandable={editor=, atlas_doc_format=, view=, export_view=, styled_view=, dynamic=, editor2=, anonymous_export_view=}}, extensions={position=165962830}, _expandable={container=/rest/api/space/DOC, metadata=, restrictions=/rest/api/content/446627872/restriction/byOperation, history=/rest/api/content/446627872/history, version=, descendants=/rest/api/content/446627872/descendant, space=/rest/api/space/DOC, childTypes=, schedulePublishInfo=, operations=, schedulePublishDate=, children=/rest/api/content/446627872/child, ancestors=, draftVersion=}, _links={editui=/pages/resumedraft.action?draftId=446627872, webui=/spaces/DOC/pages/446627872/Test+Document+DOC12, edituiv2=/spaces/DOC/pages/edit-v2/446627872, context=/wiki

Any ideas on how I can just get the plain text out of it?

[Tomislav Tobijas]

@David Stokes hmm, the best I got was by using the following endpoint:

https://your-domain.atlassian.net/wiki/api/v2/pages/PAGE_ID?body-format=storage

And you could drill down to body > storage > value but it still isn't the clean version.

I managed also to find this feature request: CONFCLOUD-83388: Add a smart value to retrieve the full body/content of a Confluence page in Automation > you can check description and suggested workaround and potentially use that (or AI summary) for what you're trying to build 🤔

[David Stokes]

Yep that’s what I did too. 

I’m going to be talking to k15t - outside of Atlassian, they know the most about Confluence, and I’m hoping they might have a way of extracting the plain text. 

I’ll let you know how I go

Thanks for your help

Cheers, David

[Tomislav Tobijas]

Sounds good @David Stokes !

@Kris Klima _K15t_ might be of some assistance here - at least to point you to the correct channel within K15t 🙂

Btw, I would appreciate it if you could share any solution (that you might find) here as well.

[Kris Klima _K15t_]

@David Stokes and @Tomislav Tobijas wow, I feel no pressure whatsoever :) 

[David Stokes]

@Kris Klima _K15t_ I'm going to catch up with Dash next week.