Database connection conundrum

Steven Mustari

I understand before asking this questions much, much, more information to get an accurate answer, but I think some one could likely provide more insight to this situation, and any additional insight would be appreciated.

Contextural information:

We have a relatively complex server instance of JIRA/JIRA Service Management. Running on Windows Server as a service with a MSQL database. To give a bit of context: ~700 custom-fields, 30-40 actively managed workflows, many scripted fields, jobs, listeners, many behaviours, and a few fragments. We have many mail handlers consuming email from various locations, as well as a Service Desk mail handling. We generate around 75k-100k issues a year in all projects. We have about 25ish applications the big ones being:

Tempo
Structure
Scriptrunner
JMWE
Zephyr
Deviniti Extensions (Bundled fields, Queues)
JIRA Misc Custom Fields
In-Mail Handler
The Scheduler

Many others, but those being the biggest impacted I'm guessing.

A bit more before the questions...

We have some custom Powershell scripts ran weekly to copy the production instance and Database weekly. It copies the production JIRA data over to a test environment, and the Production database, we insert Dev keys for everything pre-boot, and then the instance boots up licensed Dev with all production data intact.

For the second time now, we've had a failure in this process which resulted in dbconfig config copy failures and inadvertently ended up with both Prod and Test JIRA pointed to the same DB for a time. The last time this happened it was ~12 hours. The most recent occurrence of this was closer to 72 hours as it included a weekend.

The Question(s)

Luckily not much activity takes place in the test instance, I am not so much worried about the data integrity from changes happening there. My big question is the long term impact from something like this happening.

Noticeable effects that I've seen both times prior to a disconnect and reboot are: Mail Handlers fail to work properly, anything dependent on cron statements tends to fail, configuration settings from test seem to at times override production config settings.

I've spoken with Atlassian support on this, and obviously the recommendations were to rollback, and little insight could be given to the impact of the 3rd party application configurations. Does anything stick out glaringly as a long term problem once the initial cause is remediated, and everything is pointed back to where it belongs and restarted?


Thank you for any insight on this topic, again I know a lot of specifics would be needed to truly understand all the impact.

Nic Brough -Adaptavist-

There's one very simple thing that stands out to me:

>which resulted in dbconfig config copy failures and inadvertently ended up with both Prod and Test JIRA pointed to the same DB for a time.

Why in heck's name are you copying production settings to a test system?

By all means, copy the database to a test system, and the attachments, and do all the things to isolate test from everywhere else, but why are you copying the database connection to production over to a test system?

The thing that sticks out as a long term problem is "stop connecting test systems to your production database"

I know that's quite a harsh and bloody-minded attitude, but I really can't see why you would do this, or think it's a useful way to get a test system.  This is not as complex as you seem to think - just stop connecting test to live and you are ok.