Bad Confluence searchindex query performance

[Marco Leist]

We have a big problem with the performance of our confluence intranet system and suspect that the search index might be the cause:

We see search query times of ~1200ms for very simple search queries like "123" (/dosearchsite.action?cql=siteSearch ~ "123").
The system is a 15.000 user data center instance with ~5 Mio Content Objects, hosted on AWS EC2 (three c5.4xlarge nodes, each Node with 8 Cores and 16 GB RAM)

We also host a second smaller confluence instance (extranet) that only contains around 175.000 objects (hosted in our own datacenter) und here, the search request times are much faster (200-300ms).

The problem ist that many functions and macros are internally based on search queries / CQL. And therefore, the slow search has a massive impact on the perceived performance of the entire environment so that a quite big amount of our pages that contain index based macros need 10s or more to load.

The question we are now asking ourselves is whether such long query times are actually "normal" for an instance of this size, or whether there must be another cause for this.

It would help us a lot if we could get some comparison values from the community (number ob content objects, mean query times) to be able to estimate how our server performs relative to others.

Or even hints / experience values, how search queries can be made faster?

Thanks,
Marco

[Marco Leist]

Quick update:
In the meantime, I received feedback from an admin of a nearly similar sized confluence (server) environment. There, the times for simple search queries like /dosearchsite.action?cql=siteSearch+~+"123" are around 200ms instead of 1200ms.

Question: Is the search also that fast in your environments?

thanks, Marco