Out of Memory due to high usage of com/atlassian/confluence/search/lucene/TermsQuery

[ivanchou]

In our Confluence Data Center install (7.17.4), we were seeing Out of Memory errors seemingly out of the blue (we were not upgrading Confluence as other KB articles have suggested OOM errors due to reindexing upon upgrades).  The JVM runtimes were very short lived, churning through 4 GB of our normal heap size in a few hours, requiring restarts several times a day.

After reviewing heap dumps and GC graphs, we saw leak suspects like these:

1,488,688,800 bytes (33.8 %) of Java heap is used by 44,700 instances of com/atlassian/confluence/search/lucene/TermsQuery

There's not much info about this class.  But one interesting thing is it is fairly new, only recently introduced in 7.17 ( https://docs.atlassian.com/ConfluenceServer/javadoc/7.18.1/index.html?overview-tree.html
com.atlassian.confluence.search.lucene )

[Graham Twine]

Hello @ivanchou 

Can you see if there is an automated process doing something new?

Systems connecting to the REST API and doing very bad CQL searches?

[ivanchou]

There is a similar sounding Lucene OOM fix for https://jira.atlassian.com/browse/CONFSERVER-69488 in 7.19, but it doesn't mention TermsQuery search specifically.

We were able to remediate the problem by reducing the number of Max Simultaneous Searches: https://confluence.atlassian.com/doc/configuring-search-175210673.html 

I changed the value of Max Simultaneous Queries from 40 to 3.  We didn't see the problem come back, even though I don't believe any of the search activity was explicitly user driven.