How to use Confluence API to get actual data / values from Page Properties

[Jessica Malenfant]

I'd like to use the Confluence API to get the actual data entered for a confluence page tracking statements of work / contracts. I have 1 page per contracted project that uses a global template for users to enter the SOW contract details. Ineed to get the data from each page in this space that I can then pull into a BI analytics tool like Qlik or Tableau to look at projects over time, budgets, deliverables, timelines, etc. and perform calculations, etc. The Page Properties Report macro helps to see a summary of these but I need to also do analysis / report on the data.

I can't figure out what url to use to get at any of this. Here's an example of what we have now

And I can connect to the API for the space, I just can't figure out how to get the data now.

https://ourinstancename.atlassian.net/confluence/rest/api/space?spaceKey=OURKEY

 

 

[ekarlssonjohan]

Since I spent about 8 hours to figure this out I thought I might post an example here just in case someone else stumbles on the same issue.

This REST call will retrieve all page properties for each page where pages have the label: label=mylabel.

http://localhost:8090/rest/masterdetail/1.0/detailssummary/lines?cql=label%3Dmylabel&spaceKey=MYOWNSPACE

[Ajay Verma]

Hi 

am using your url and getting the below error

{"errorCode":500,"errorType":"RENDERING:MACRO","errorMessage":"Unable to locate pages for Page Properties Report. Details: CQL string is \"(label=projectinfo) and macro = details order by lastModified desc\", error message is: \"Not permitted to use confluence : null\""}

[jmakarlsson]

Hi,

actually I think I only manged to get it to work in a browser if I remember correctly. Also you need to be logged in to confluence. 

[Ajay Verma]

Now I can read the data in jason thanks to you . but can you please confirm I have 10 columns in my macro page but I am getting only Tital from there . how can I get rest of the columns ?

[jmakarlsson]

Not sure I understand, what does your url look like? 

However, maybe it works if you add page size and index? Like: 

https:// localhost:8090/rest/masterdetail/1.0/detailssummary/lines?cql=label%3Dmylabel&spaceKey=myspace&pageSize=30&pageIndex=0&headings=myheading

[Ajay Verma]

HI

could you please help me to correct the below url is it correct or i missed something.

https://mysite/rest/masterdetail/1.0/detailssummary/lines?cql=label=projectinfo&spaceKey=project&pageSize=30&pageIndex=0&headings=Projektledare, Projekttyp, Fas, Overall, Budget, Tidplan, Kundnöjdhet, Lösning, Uppdaterad

also am not getting all records some records i am missing what I can do to get all records

[jmakarlsson]

I tried your url with multiple headings and it works. Any difference if you remove the spaces between the "headings"? Could the swedish characters casue any problem? 

Note: that you might get HTML as part of the key (heading) value.

Regarding number of records, not sure what you mean. But what happens if you increase page size to 100? 

[Ajay Verma]

Hi

Thanks for reverting . am not getting all records some are missing . like RI Förvaltning 2020 . already increase the page size to 100 below error I have recieved

{"currentPage":0,"totalPages":1,"renderedHeadings":["Acceptansdatum"," Ansvarig säljare","Budget","Fas","Hemlig handling","Hemlig uppgift","Individuell sekretess","Kund","Kundens projektledare","Kundens projektsponsor","Kundnöjdhet","Kundnummer","Lösning","Overall","PUB(GDPR) avtal","PUL","Projektledare","Projektstart","Projekttyp","Resurser","SUA-klass","System","Teknikinformationskontakt","Tidplan","Timpris","Uppdaterad","Ursprungligt acceptansdatum","Verksamhetskonsult"],"detailLines":[],"asyncRenderSafe":false}

What I can do to get all records

[Ajay Verma]

I have checked and found one issue when I changed page index to 1 then I received the missing records but other one disappear . 

could you please confirm whats the mean of page index here  

[jmakarlsson]

I guess you need to include "RI Förvaltning 2020" in the url. Or could there be a problem having spaces in the key/heading?

Regarding page index. If you have 1000 pages and page size is 200 then page index=0 will show records 0-200 and page index=1 will show 201-400 and so on. 

[Ajay Verma]

I think index making a problem here . I have already tried to remove pag size and page index but still the same issue . 

any idea what i can do now to getting all records

[jmakarlsson]

No idea actually. I don't have that problem. I set &pageSize=500000000&pageIndex=0 and I am able to get all pages. 

[David Memmer]

even though you change the pagesize you cant make it as big as you want. There will always be a limit (at 500 i think) and you need to start paging

[Ajay Verma]

When I change the page size like 70 or 100 its give me below error

{"currentPage":0,"totalPages":1,"renderedHeadings":["Acceptansdatum"," Ansvarig säljare","Budget","Fas","Hemlig handling","Hemlig uppgift","Individuell sekretess","Kund","Kundens projektledare","Kundens projektsponsor","Kundnöjdhet","Kundnummer","Lösning","Overall","PUB(GDPR) avtal","PUL","Projektledare","Projektstart","Projekttyp","Resurser","SUA-klass","System","Teknikinformationskontakt","Tidplan","Timpris","Uppdaterad","Ursprungligt acceptansdatum","Verksamhetskonsult"],"detailLines":[],"asyncRenderSafe":false}

[WW]

What you're getting as a result is not an error (I can't read the human language it's in, though) but it looks like your result is just empty.  That's this part:

"detailLines":[]

If you read my last post, sometimes I have to figure out what the total number of results is, then enter that as the page size.  No idea why, and it only seems to be for certain fields.  This may or may not be your problem, though.

You should also make sure your cql is good.  I think you solved the error issue with the spaces, but what you're actually searching for may not have any results.  Like, you might not have any pages with the projectinfo label.

Try simplifying the cql to something that has no special characters just to see if you get any results such as:

https://localhost:8090/rest/masterdetail/1.0/detailssummary/lines?cql=label%3Dprojectinfo&spaceKey=project&pageSize=30&pageIndex=0&headings=Budget

You can also try leaving off the pageSize and using the default to see if you get any results such as:

https://localhost:8090/rest/masterdetail/1.0/detailssummary/lines?cql=label%3Dprojectinfo&spaceKey=project&headings=Budget

(I think you may have done this already, but without a heading)

If you don't get any results from the above or get empty results again, then make sure:

• the spelling of your label is correct and that you actually have pages with that label
• that you have a table with 2 columns (for vertical tables) or 2 rows (for horizontal tables) actually inside of a Page Properties macro on the page.  You'd be surprised at how often they're setup wrong
• there are values in the correct column/row for the pages
• the column name is spelled correctly (Budget should be simple enough, though)

If running it with just Budget works, then your problem is most likely the special characters.  You should encode your URL before sending it.

In any case, EVERYONE ON THIS THREAD, if you haven't already, please go and VOTE for this issue CONFSERVER-37890.  It basically asks to increase or eliminate the current limit of 500 results.

[Mousuma Roy]

I am using  this https://ourinstance.com/rest/masterdetail/1.0/detailssummary/lines?spaceKey=myspacekey&cql=label%3Dmylabel

I am getting this. 

{

"currentPage": 0,

"totalPages": 0,

"renderedHeadings": [],

"detailLines": [],

"asyncRenderSafe": true

}

my page with this label has 10 macros.
any idea why I am getting this

[WW]

You might not be authenticating properly or the label might be misspelled. It's hard to tell from the given results.

I'd need more details like what code you're using to make the call, if you're on Cloud or Server/Data Center, etc.

[Luis Gonzalez]

Hi!

The API answer seems to paginate, but there is no link to the next page... nor parameter to access a page in particular.

Trying to find a workaround, if &pageSize=99 added to the url, detailLines returns empty instead of the 30 detailLines that returns without that parameter.

Any suggestion? 

Thanks in advance,

Luis

[WW]

I was having this same issue, but I figured it out.  You should see the totalPages in the first part of the results.

{"currentPage":0,"totalPages":17, etc.

To see more pages, add &pageIndex=1 to the request to get the next page. 

If there are 17 pages, &pageIndex=16 would be the last page (0 would be what's included in the first page).

Here's what it would look like to see the second page, using the example given by Robert (above), with the additional pageIndex part:

/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type=page&spaceKey=MYSPACEKEY&headings=key1,key2,key3&pageIndex=1

[David Memmer]

Adding &pageSize to the query works for me. Has there been an update ?

[WW]

It's weird.  Sometimes setting the pageSize to 1000 works, and sometimes, it will only work if I set it up to 62, which is the exact number of records that exist for the cql I put. 

I think it depends on what my headings are.  In particular, it doesn't like the heading "Compliance Plan."  If I change the spelling of it, it gets the data and leaves that field blank.  If I leave that heading in there, it returns no records.  I've tried headings that are longer and shorter, and they work.  I have no idea why it doesn't like that field.

Like I said, weird. 

[WW]

And, FYI, this API must tie in to the same code the page properties report macro does because there is the same issue of a limit of 500 results regardless of how many pages they display on.  To get more than 500 results, you'd have to do multiple calls/searches.

[Robert Reiner _smartics_]

We do not use this API since we use our document properties. :-)

But this REST call should work

https://example.com/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type%3Dpage&spaceKey=MYSPACEKEY&headings=key1%2Ckey2%2Ckey3

 Or with cURL

curl 'https://example.com/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type%3Dpage&spaceKey=MYSPACEKEY&headings=key1%2Ckey2%2Ckey3' | jq .

[Jessica Malenfant]

Thanks so much. I actually looked at your products but it looks like your tools are only for server and not cloud - is that true?

As far as your example REST call:

I swapped out the example.com for our site and spaceKey for ours, but can’t figure out what the headings are equivalent to?  How do we drill down to the pages and its child pages? I tried w/ our page name like this:

https://example.com/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type%3Dpage&spaceKey=PDP&headings=Project+ABC+Contract%1  

error:

Unable to locate pages for Page Properties Report. Details: CQL string is \"(type%3Dpage) and macro = details order by lastModified desc\", error message is: \"Could not parse cql : (type%3Dpage) and macro = details order by lastModified desc\"  

Then tried a few variations that also failed:

  https://example.com/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type%3Dpage& spaceKey=DP&headings=keyContract+Summary%1  

https://example.com/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type%3Dpage& spaceKey=DP.Contract+Summary+-+Open+Contracts&headings= Project+ABC+Contract%1

Any insight you have (to this very novice coder!) is greatly appreciated!!!

Thanks!

[Robert Reiner _smartics_]

Thank you for checking out our Web API! Indeed it is only available on Confluence server.

Sorry, it seems that I messed up coping the URL.

The headings are the keys to the page properties.

This works for me (I snipped off the protocol/server part) using a web browser:

/confluence/rest/masterdetail/1.0/detailssummary/lines?cql=type=page&spaceKey=MYSPACEKEY&headings=key1,key2,key3

Here the first characters of the result I get:

{"currentPage":0,"totalPages":17,"renderedHeadings":["key1","key2","key3"],"detailLines":[{"id":70189634,"title":"Example Page with Page Properties","relativeLink":"/display/MYSPACEKEY/Example+Page+with+Page+Properties","details":["value1","value2","value3"],"likesCount":-1,"commentsCount":-1}

I assume that you need to check out the CQL documentation to learn how to navigate through the pages. I would expect that selecting on pages (with something like key1=value1) should be possible, but unfortunately I do not know the syntax (due to the lack of documentation). :(

It seems that the REST API Browser is not available for Cloud. This would be the UI (as an overview over the supported parameters):

[Jessica Malenfant]

Thanks so much - I'll try this out today. Any thoughts about providing a cloud solution in addition to server? :)

[Robert Reiner _smartics_]

Hi Jessica,

I'm happy if I could give you some links to start with. Hope you figure out how to use the REST API despite the missing docs quickly ... ;)

Thinking about Cloud is what we do, but unfortunately this is still a very, very long way to go. Thank you for your feedback!!

Cheers,

Robert

[Robert Reiner _smartics_]

Is this what you are looking for?

How to retrieve metadata from Page Properties using REST API

Unfortunately I have not found any useful documentation on how to use this API.

EDIT: Space Key is required and headings allows to specify the properties you are looking for.

This seems to be the source code on Bitbucket.

[Jessica Malenfant]

Thanks! I actually just found that one! Bummer about no documentation! 

Did this end up working for you? I keep geting  a 404 error when i try:

https://OURINSTANCE.atlassian.net/confluence/rest/masterdetail/1.0/detailssummary/lines/api/space?spaceKey=OURKEY