Stories about detecting in my Atlassian Confluence instance bottlenecks with APM tool [part 3]

[#1]

Gonchik, a lover of APM (application performance monitoring) tools, in particular Glowroot, is in touch. This is the second story related to observing Confluence with Glowroot.

Since the parsing was not limited to work between the DBMS and the application, going to the Queries tab and sorting by the column “Average execution time”, I saw an interesting pattern, which is shown in the screenshot below.

Yes, it is in the AO_7B47A5_EVENT table that occasionally INSERT is slow when a large number of users open pages in Confluence. And since there is a prefix AO_ *, it is an indicator of tables created by ORM Active Objects. As a rule, these are plugins installed on the platform. Here Google quickly pointed to the next ticket https://jira.atlassian.com/browse/CONFSERVER-69474, which indicated that the culprit was the Confluence Analytics app. Searching in Managed Apps, we find it, after setting All Apps in the search scope. Voila! Having found the application, we see that it is not licensed, according to the screenshot.

Since our installation is licensed as Server, not Datacenter, you can safely disable the application, since it does not carry a payload due to the lack of a corresponding license.

The situation is not very acceptable, so I sent a request to the vendor about the need to present metrics and rules for the retention policy of analytics logs in a database full of asynchronous analytics work. As a result, after clearing the records in the table, the content delivery accelerated before our eyes.

We conclude that it is important to verify slow queries and watch their logs (at the time of the audit they were not available for analysis). This helps a lot to provide information to vendors so that improvements can happen. For example, the analytics extension in the latest version of Confluence works much better and does not affect the system as much, although there is a problem when working with attachments.

To summarize, we analyze the main business metric, namely the user experience, and get feedback from the business customer: “There were no complaints.” However, we have something to do, as a rather large backlog of changes to improve the system has been collected.

Thank you for reading to the end, I will be glad if these situations were interesting, then I will continue to share similar stories of using tracing and search tools.

I am always glad to have your questions, and also to see you in the Telegram chat of the Atlassian community.

Have a nice day, Gonchik.