com.atlassian.sal.api.net.ResponseException: Can not retrieve jira chart image

[Richard Duffy]

Hi All

Background Info:

Jira version - 8.16

Confluence version - 8.11.1

Environment - Datacenter with 1 load balancer and reverse proxy

Cluster - 2 Jira nodes

*This issue is in my prod and sandbox environments*

Since we upgraded from Jira 8.11 to 8.16 we are experiencing random issue reading Jira charts macro in a confluence page with following error, some charts will load and some wont. Refresh screen and some others may load and some which were loading may error

com.atlassian.sal.api.net.ResponseException: Can not retrieve jira chart image 

Looking at Jira logs I can see 

2021-05-17 11:01:34,632+1000 http-nio-8081-exec-25 WARN anonymous 661x41935x1 - 10.50.0.249,10.50.0.249,10.20.50.37,10.50.1.154 /charts [c.a.j.charts.jfreechart.DisplayChartServlet] File not found, returning 404 (filename='jfreechart-onetime-620730146466603863.png').

However looking at localhome/cache/charts I will be able to see this chart reference on 1 of the 2 nodes

This issue seems to be related to  JRASERVER-67481   or Inserting JIRA chart into Confluence Page results in a ResponseException and chart does not display  however I notice my files charts/cache files are not being written to the EFS shared home location, instead my cache files are being written to localhome/cache/charts on each node

Test - Create new charts folder with correct permissions in sharedhome/cache/charts and on each node localhome/cache/charts,  Load the Confluence Page

Result - No activity in sharedhome/cache/charts

Files are written to localhome/cache/charts across the 2 Jira nodes and they have following permissions "rw-r" and owner same user who installed jira, which I believe should be sufficient?

Due to the behaviour of the page where some files load and some dont, and when I refresh it may reverse it makes me think that each node can only read the charts in its own localhome directory. Also if I drop my clutser to 1 node all charts will load

After downgrading to 8.11 the files are not being written to my sharedhome again and the issue is resolved. 

Anyone else experience this behaviour in a clustered environment on Jira 8.16?

Thanks

Richard

[Richard Duffy · Accepted solution]

This issue was been matched with Atlassian bug https://jira.atlassian.com/browse/JRASERVER-67481

See updated comment with new behaviour 

[Richard Duffy]

Two Workarounds:

• Reduce cluster to 1 node with a clean restart of that 1 node

• Switch chart macro for Jira pie chart macro (save the jql as a jira filter for input)

[Tamas Bela]

Thanks @Richard Duffy  You comment helped me a lot to understand the issue. Basically what we are facing is an ugly bug, and I'm really surprised it didn't attracted more attention. So the one-time pictures of the charts are generated in the local caches directory of one Jira Datacenter node in the cluster, and that node is chosen randomly, independently from the fact that which node was contacted originally via the Application Link from Confluence. In other words, if you have a 3-node Jira cluster, you have only 33% chance that the pie chart will be displayed correctly in Confluence. Otherwise, you'll get a nice 404. (tested with: Jira 8.16.1 / 8.17.0 <-> Confluence 7.12.2)

I was able to workaround this by a simple masterstroke. Just replace your local ../caches/charts directory with a symlink, pointing to your shared home charts, like /nfs/jira/caches/charts/. As all files are generated with a unique name, it won't cause any concurrent write problem. And they will be removed afterwards automatically. Bob's your uncle.

I'm reposting this to the bugtracker page too to let more people know about it.