Kubernetes deployment -- suddenly getting "Confluence tables already exist" message on startup

[Eric Cleek]

Situation is this:

A Jira SM and confluence (data center) instance has been running on AWS EKS utilizing AWS RDS for a DB and EFS as persistent volumes for both JSM and the Confluence as shared-homes as both were running clustered.  They are deployed via helm.

A few days ago, I had to deploy new helm charts that had no effect on the jira or confluence charts at all (it was simply adding another application)  After I redeployed all the apps, confluence can no longer recognize its own DB and shows the typical "Confluence tables already exist" message on startup. 

While this is not officially a "production" DB per se, it does contain a lot of data that we cannot lose.  

I've attempted to restore from backups with both the EFS and DB and the issue remains.  I have looked all over the place for a solution to "Confluence tables already exist" and I've found nothing that works and most are solutions offered for non-k8s environments which are either impossible to replicate in a containerized env, or beyond my skills.  I've been at this for more than a week.  Of all that I could try nothing has worked, including:

1. Full restore (as mentioned above)

2. "Installing" onto a new db and then switching back the original DB.

3. Full restore with a much older backup as a troubleshooting step

4. A hundred other small things that I can't even recall.

So, while I'm looking for general help, I'd also love if someone could tell me what exactly couples the home and shared-home to the DB, and why there is no guidance on how to fix this.  Given how common this issue has been in my own work, it seems like there would be readily available answers to this question, but, again, near nothing with an actual solution exists online. 

Edit:  I am also seeing a number of "Spring Application context has not been set" errors in logs.  I see no other errors to indicate why.

[Yevhen]

@Eric Cleek most like you have lost data in your shared home which contains confluence.cfg.xml. Now when the confluence starts, it thinks it's in a setup mode (as it it's a new installation). Confluence connects to the DB and wants to initialize it but discovers that the db has existing data in it.

[Eric Cleek]

The shared home is what was backed up completely.  Nothing is missing from the day it was working.  

---

<?xml version="1.0" encoding="UTF-8"?>

<confluence-configuration>
<setupStep>complete</setupStep>
<setupType>cluster</setupType>
<buildNumber>9012</buildNumber>
<properties>
<property name="access.mode">READ_WRITE</property>
<property name="atlassian.license.message">***License***</property>
<property name="confluence.cluster">true</property>
<property name="confluence.cluster.authentication.enabled">true</property>
<property name="confluence.cluster.authentication.secret"> ** Auth Secret** </property>
<property name="hibernate.setup">true</property>
<property name="jwt.private.key"> ** Private Key ** </property>
<property name="jwt.public.key">MIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA49HNabm5n2+owvBZO6vfppCTigHQzQFMPTNCR2Zd2Wj3NJ/GwHB7qfBrobFjcM7fr1U0xDrP6JQX67hAzcA7XiUfgZyYHic2f48GSRQlrvYzZQhguEqeecYR1QYQLtBXa0j7ztOk3+pG72nM/CMEQ+bxggDlb/xAZGLuG3EbweL5vb6/p/+5KJZZOM0FXOVIewdX/ud9TtCx7CuVMqSCP+GRfDB9xyk8VKdVLI2w3vHE9d/mk1SiHl6D8bslWOeCznT5In86XvDlIm0LLsPRF3uHNwseN73QPOXppezjTvBrU6ucmWCPxzgcBkWVLWqBIIdFbJ8b2Y8Zds1nEE4Owp/bDnA+YBsg2G9sBeZ+gS2QUUrFSN3tLfJICbA4wQoQ2DFIIB7uiDhQlnIyBoJBB5wpCOUECCPDu9dcbc4D0OVmu3+nb/QnwzzpobE/DQ3CWYTdEhuPezqeQ9natfuouSYCGkOOaOiYFzGSyDOMHHDTBPkk3kriDMprv1VNj4uTAgMBAAE=</property>
<property name="lucene.index.dir">${localHome}/index</property>
<property name="synchrony.service.authtoken">*** Auth Token </property>
</properties>
</confluence-configuration>

---

I do understand why this is happening, I'd like to know how to fix it.  Does atlassian really not have a solution to such a simple migration issue?  I've looked over every doc I can find.

[Yevhen]

How does your confluence.cfg.xml in local-home look like? Also, is clustering enabled in your helm values?

[Eric Cleek]

@Yevhen 


<confluence-configuration>
<setupStep>setupcluster-start</setupStep>
<setupType>custom</setupType>
<buildNumber>9104</buildNumber>
<properties>
<property name="access.mode">READ_WRITE</property>
<property name="atlassian.license.message">**license**</property>
<property name="attachments.dir">${confluenceHome}/attachments</property>
<property name="confluence.database.choice">postgresql</property>
<property name="confluence.database.connection.type">database-type-standard</property>
<property name="confluence.setup.server.id">** server ID</property>
<property name="confluence.webapp.context.path"></property>
<property name="hibernate.connection.driver_class">org.postgresql.Driver</property>
<property name="hibernate.connection.password">** db pass **</property>
<property name="hibernate.connection.provider_class">com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider</property>
<property name="hibernate.connection.url">jdbc:postgresql://rds.url:5432/cfl_db</property>
<property name="hibernate.connection.username">cfl_user</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.PostgreSQLDialect</property>
<property name="hibernate.hikari.idleTimeout">30000</property>
<property name="hibernate.hikari.maximumPoolSize">100</property>
<property name="hibernate.hikari.minimumIdle">20</property>
<property name="hibernate.hikari.registerMbeans">true</property>
<property name="jwt.private.key">** Private Key **</property>
<property name="jwt.public.key">MIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAuLDjJmvBq3M0zLa5Fs4iWjx+nD40e40FeCa2ZiUPrCMESAUL0/LaHb2X2HxdRkCfpwTGf1/PRs45ZKLb6yfXJV8HncRuGrpLTZsVj1vaZyBHGYo3FXCHbBJEY2luc2p/Y3xvbEqGmWKKQyB0REDpLB3H2Dz/pRcjdqnaK4Cke5kKKQ7SgShTta/DCOp6iPOCABH/H6Q35/huWCDoxkfJy5jWzCFzWebjRKLPQeMPwzHBdNBSBhLdRGTSpg98tFo5hI8DoC2nFhBQsNnILShT98tLSbp2LAS4XAlippoy2c2bgKdfvjacWRKzEANOCkqD1T8KXD4udyFBbhTqBzXWjUxosChLNB7aoCVDhdqCrYhb1SzKv8rJvhmS3THxo92erFkmVw+C/6u5I/joJoayWzyyoX7tU8iGyDywDlMF1Jil227cJk0MXkoBJbUSMTN2vVQGhO2xXpQrKuZQUVH/3xiOLCrtZeQ0R6YimSrBtmj10j8ZaP1F5pYu7hBUm94TAgMBAAE=</property>
<property name="lucene.index.dir">${localHome}/index</property>
<property name="struts.multipart.saveDir">${localHome}/temp</property>
<property name="synchrony.encryption.disabled">true</property>
<property name="synchrony.proxy.enabled">true</property>
<property name="synchrony.service.authtoken">cd44e23dbbf2d8ca5081f82b62261bbc</property>
<property name="webwork.multipart.saveDir">${localHome}/temp</property>
</properties>

This was with clustering disabled.  I can potentially get a version of it with enabled if needed.  

Also, I know this is not at the right setup step.  I have tried manual modification but, but k8s seems to overwrite the file even with forcecfgupdate turned off. 

[Yevhen]

@Eric Cleek you must turn on clustering in helm values and keep forceCfgUpdate=true. Right now, Confluence is not looking for confluence.cfg.xml in shared-home (which has setupStep=complete).