Data Center Love Month: OpenSearch for Confluence

[Anubhav Dutt]

Hello Data Center Community,

Happy Data Center Love Month!

I’m Anubhav, a product manager in Atlassian Data Center and I am excited to announce a new enhancement in Confluence designed to address the challenges of slow search performance, particularly with increase in usage and growth in instance size.

Confluence uses Lucene's search library for powering search as default. Lucene, while a robust low-level search library, is not inherently designed to be scale-resilient. This limitation of Lucene becomes more apparent in large instances.

Enter: OpenSearch

OpenSearch is a popular enterprise-grade dedicated search platform that creates a powerful search capability based on distributed processing architecture backed by the robustness of Lucene’s search library.

With the latest Confluence Data Center 9.0 release we now support OpenSearch if you opt-in to switch from the default Lucene based search implementation.

Search in Confluence is now faster:

Our internal performance benchmarking results show OpenSearch is ~4.5x faster than a comparable Lucene-based search and this performance gain only widens with large dataset.

The results:

We tested this on an internal 3 node Confluence instance for benchmarking a 36GB index using Lucene(8 CPU, 32GB RAM) against OpenSearch(2 CPU, 16 GB RAM).

Its not just search that benefits from this feature, below are several reasons for why we believe this is a game changer when it comes to overall Confluence performance experience for our large enterprise customers:

• Better performance - Search is faster and other functions that depend on Lucene access for e.g. page moves, may experience gains

• Scale with confidence - As your organization grows in size and scale, take advantage of OpenSearch’s horizontal scaling capabilities and let Confluence grow with you.

• Reduced overhead - Simplified index management lowers overall Confluence resource consumption.

• High availability - Search without interruption on clustered OpenSearch instances during reindexing.

Our exploration and thorough tests have confirmed that OpenSearch can significantly improve scale related performance issues, however we also understand that switching to OpenSearch may not be immediate for some customers.

OpenSearch is not the default search platform for Confluence.

To begin to use it, you will need to setup and configure it as first-step.

Visit OpenSearch configuration guide containing links to OpenSearch resources.

Search is a core function for any enterprise knowledge management solution and the wide adoption of Confluence has meant that we keep enhancing and improving the Confluence Search features for our customers. In the recent past we introduced space re-index with Confluence DC 8.4 to ensure users can promptly access the most up-to-date content and modifications in real-time fostering a seamless and uninterrupted collaboration experience among teams.

For more details on OpenSearch refer to Confluence DC 9.0 release notes.

To enter for your chance to win Data Center exclusive swag*, leave us a comment or question in the comment section below.

*At this time, swag is available for US shipping only. Winners will be notified at the end of the month.

[Gonchik Tsymzhitov]

To be honest OpenSearch looks very promised and provided in AWS :) 

[Eddy G]

Very much looking forward to improved search performance in Confluence. 🙂

Quoting from the OpenSearch docs:

There are many ways to design a cluster. The following illustration shows a basic architecture that includes a four-node cluster that has one dedicated cluster manager node, one dedicated coordinating node, and two data nodes that are cluster manager eligible and also used for ingesting data.

Will Atlassian be providing any recommendations (or at least an example or two) for how to size and configure an OpenSearch cluster (in terms of CPU, memory, disk space) for ”good” performance based on the size of the Confluence instance? (Especially for customers running on-prem and not AWS?)

Thanks!

[Stephen Garber]

This is great, search has been a pain point for confluence for a while. It can be slow and produce low quality results.

Moving pages in bulk can also take a while.

It's a great idea to move searching off the main instance!

[Anubhav Dutt]

Thanks @Eddy G  for your comment and sharing your excitement !!

Regarding the size and configuration we have tried to address that here.

However this is based on Amazon AWS setup that we tested with primarily. My suggestion would be to setup a similar configuration on-premise at the minimum and scale up or down depending on specific usage.

Hoping that helps.

Anubhav Dutt

PM - Atlassian DC

[Eddy G]

I'm not sure showing managed OpenSearch nodes really helps folks running on-prem. Those instances, and their OpenSearch settings, are all preconfigured by Amazon.

It would be much more helpful to show the details of each node for several OpenSearch configuration options (based on the number of users, size of the Confluence instance, etc.) when OpenSearch is run in an on-prem Kubernetes cluster, for example.

[Kristin Lyons]

Reduced overhead - Simplified index management lowers overall Confluence resource consumption.

I love this, we are always looking for ways to ensure we are scaling both Jira and Confluence properly so having reduced overhead is huge for us!

[Dave Liao]

I’m here for the swag!

Just kidding, I’m actually curious about how much OpenSearch can help my users… 🔎