Hi All
Background Info:
Jira version - 8.16
Confluence version - 8.11.1
Environment - Datacenter with 1 load balancer and reverse proxy
Cluster - 2 Jira nodes
*This issue is in my prod and sandbox environments*
Since we upgraded from Jira 8.11 to 8.16 we are experiencing random issue reading Jira charts macro in a confluence page with following error, some charts will load and some wont. Refresh screen and some others may load and some which were loading may error
com.atlassian.sal.api.net.ResponseException: Can not retrieve jira chart image

Looking at Jira logs I can see
2021-05-17 11:01:34,632+1000 http-nio-8081-exec-25 WARN anonymous 661x41935x1 - 10.50.0.249,10.50.0.249,10.20.50.37,10.50.1.154 /charts [c.a.j.charts.jfreechart.DisplayChartServlet] File not found, returning 404 (filename='jfreechart-onetime-620730146466603863.png').
However looking at localhome/cache/charts I will be able to see this chart reference on 1 of the 2 nodes
This issue seems to be related to JRASERVER-67481 or Inserting JIRA chart into Confluence Page results in a ResponseException and chart does not display however I notice my files charts/cache files are not being written to the EFS shared home location, instead my cache files are being written to localhome/cache/charts on each node
Test - Create new charts folder with correct permissions in sharedhome/cache/charts and on each node localhome/cache/charts, Load the Confluence Page
Result - No activity in sharedhome/cache/charts
Files are written to localhome/cache/charts across the 2 Jira nodes and they have following permissions "rw-r" and owner same user who installed jira, which I believe should be sufficient?
Due to the behaviour of the page where some files load and some dont, and when I refresh it may reverse it makes me think that each node can only read the charts in its own localhome directory. Also if I drop my clutser to 1 node all charts will load
After downgrading to 8.11 the files are not being written to my sharedhome again and the issue is resolved.
Anyone else experience this behaviour in a clustered environment on Jira 8.16?
Thanks
Richard