Confluence DC with HA database

[Bert Kelder]

Hi,

I just commented on an issue: https://jira.atlassian.com/browse/CONFSERVER-20277 which is about making Confluence DC (and Jira for that matter) High Available on Database level, hosting in own datacenters / cloud (not in AWS/Azure)

I'm looking for teams / companies having actually done this, if they would be willing to share their setup and experience

[Michael Yecies]

Hi Bert, 

For my company I architected and deployed an HA PostgreSQL database consisting of two Linux servers each running PostgreSQL in a Master - Hot Standby architecture.  This architecture has been in place for 4.5 years and has provided us with 100% uptime aside from very rare service windows.

There is automatic failover from Master to Standby and the data is automatically replicated asynchronously from Master to Standby.

The method we use can be expanded to allow for cascading failover to additional Hot-Standby PostgreSQL servers; as many as you like. Ex. Master -> Standby 1 -> Standby 2 -> Standby 3 -> ... Standby n

We also make use of a continuous "streaming" backup of all database transactions with a 4 day rolling recovery window.  This Point In Time Recovery (PITR) system allows us to restore the database to any point in time during the 4 day window.  We also make use of other several backup methods for the database; storage is inexpensive... the content is not.

This PostgreSQL deployment has been fantastic for us and as stable and performant as they come.

This system is on-premise on hardware and VMs I architected and deployed and has supported as many as 3000 active users (we actively manage user accounts and de-activate non-active users) with peak loads of over 1000 simultaneous user sessions (logged in users using the interface).

In addition to the model I have deployed there are also methods to deploy a PostgreSQL "multi-master" or Active-Active DB cluster.

If you would like to discuss further let's get in touch directly.

Best,

Michael Yecies

Architect and Team Lead - DevOps and Collaboration Systems

linkedin.com/in/michaelyecies

[Danny van den Berg]

Dear Michael,

Thank you for your response and clear explanation.

But what about when one of the masters fail and PostgreSQL will failover to the slave with a different IP? Because Confluence is not able to make these switch when the hostname of ip changes. We would also like to have this setup, but our current private cloud doesn't support floating IP.

Looking forward to an answer.

Kind regards,

Danny

[Michael Yecies]

Hi Danny, 

We use a VIP (virtual IP) for the database servers. Each DB has its own fixed IP address then inbound application connections to the DB go to the VIP which is routed on the master DB. 

When we failover the VIP is switched via script from master DB to standby DB (now new master DB) as a part of the failover process.  The now old master DB is still running but is no longer accessible to the applications and is not joined to the cluster and is waiting triage, fix, and then manual redeployment to the cluster as new standby DB and then we manually resync the DB data from master to standby and restore async streaming replication thereby restoring the cluster to full HA mode (... in my 2 server model. In a 3 server model the cluster would still be in HA mode unless 2 servers failed; extrapolate to n-number of servers per your risk management / paranoia). I should note my DB server deployment has 100% uptime for 4.5 years aside from very rare planned maintenance (basically upgrades to the HS cluster managment software, Ubuntu server VM upgrades, or PostgreSQL upgrades).

Applications connected to the DB cluster only see the VIP and don't know what the underlying DB architecture is. From the perspective of the applications there is only a single DB connection.

We don't have any 3rd party involved as the VIP lives on only the master DB server since there is only one master at a time.  Example: something like this: https://linuxconfig.org/configuring-virtual-network-interfaces-in-linux

We do have a DNS address mapped to the VIP for convenience and "portability" in case we ever have to move locations.  In general, we seek to have all inter-server networking use DNS addresses where possible rather than using "hardcoded" IP addresses.

Seems like you'll need to work with your data center provider (aks "cloud") to determine how you can get set up with a VIP.  Or move to another provider :-0

Best,
Michael Yecies

Architect and Team Lead - DevOps and Collaboration Systems
linkedin.com/in/michaelyecies

[Danny van den Berg]

Hi Michael,

Thank you for your response.

You infrastructure sounds like music in your ears. This is how we also would like to setup it up, but we don't have the VIP available within our private cloud. We tried already Pg-Pool-II, but still without a VIP you won't have full HA in place.

Anyway it would be nice to have a talk soon about this kind of issues!

Thank you for sharing! Have a good one!

Kind regards,

Danny

[Michael Yecies]

Hey Danny,

We installed Pg-Pool-II however we replaced it with repmgr.

We also use barman for PITR.

Cheers,

Michael

[Michael Yecies]

Danny, 

You can find me here linkedin.com/in/michaelyecies for any communication off this board.

Michael