Entire JIRA MySQL database suddenly gone... no cause known.

Hello,

One day I'm using JIRA, the next I get database not found errors.  I checked and it appears my database has vanished. Entirely, and suddenly.  I don't have backups because I actually thought I installed a plugin to do that - turns out I didn't (at least on this installation).  The data isn't absolutely vital by any means, but it's certainly a pain in the ass to lose it.

I'm curious as to how it's possible the database has vanished.

Setup: MySQL server running locally with a db called 'jiradb'.  The MySQL Server was still running without any apparent problems, just the JIRA database (and corresponding MYI etc. files) are not there. I actually can't find any trace of them.

You can see how sudden it was from the logs. I've shared them on Dropbox here:

https://www.dropbox.com/sh/15a4ry6v2rzj6hn/AACGXyKkEyMaeOBuXHMF4aOqa?dl=0

If you check atlassian-jira.log, and find the first occurrence of "2016-04-08 15:56:12,190" you will see the point at which the DB was gone.

The security log (also in the shared folder, atlassian-jira-security.log) looks to shows some activity from Russian and Chinese IPs that I do not recognise (118.193.12.194, 130.193.51.39, 141.8.132.59)

Is it possible someone has got in and simply deleted my database in such a clean way?

I expect my data is long gone at this point, but any help would be greatly appreciated.

 

*Update*: Just used a recovery tool to check the MySQL dictionary file, and indeed I'm not going mad - the database was there with what's look like a ton of tables.  So something has entirely and cleanly wiped it!  This is extremely odd...

 

8 answers

This widget could not be displayed.
Steven Behnke Community Champion Apr 09, 2016

Were you upgrading JIRA or plugins? How was your database secured? Did you have a backup policy in place of any kind? Are you on AWS and using a snapshot or backup policy? Perhaps your provider has it's own back up system? Do you have any MySQL logging to provide?

This widget could not be displayed.
Steven Behnke Community Champion Apr 09, 2016

Without MySQL logs you really aren't going to get any answer.

This widget could not be displayed.

I wasn't doing anything with JIRA.  I just attempted to load a page in a new session and got errors, which led me to this.

The database was only secured via the JIRA login.

As mentioned, no recent backups, it's a non-AWS VM with a self-managed MySQL server - no way to get the DB.  I'm trying low level recovery tools now.

I'm more interested in knowing how this could be possible really. We only had error logging on the MySQL DB (which shows nothing), so looks like I'll be out of luck knowing what happened then.

 

This widget could not be displayed.
Daniel Wester Community Champion Apr 09, 2016

Where is your JIRA home directory? If it gets wiped out and JIRA is restarted you could see this.

This widget could not be displayed.
Steven Behnke Community Champion Apr 09, 2016

I've just never heard of anything even remotely close to this happening.

You mentioned in your edit, you see tables now. Is it not possible to connect to, or can you verify you don't see custom fields, issue data, the like?

This widget could not be displayed.
Daniel Wester Community Champion Apr 09, 2016

It can happen depending on the is hosted on and where you place the home dir. If you make the home dir in /tmp on a vanilla Ubuntu installation it usually gets wiped at reboot. When JIRA comes up it will then ask to be set back up again. I can't remember the behavior for JIRA7/6 but for older versions it would more than happily help you erase the database as you're setting up the instance again.

This widget could not be displayed.

Most versions of JIRA will say "hey, there's stuff in here" if they go through a setup cycle when connected to an existing database.  If that stuff looks Jira-like, they'd ask you if you're sure you want to kill it

If you did not go through a setup process when this happened, then you've either changed the database connection settings to a database that is not a JIRA one (or doesn't exist), or you have deleted the database.  It's not a JIRA problem really, as JIRA won't do that.  Something has moved or deleted your database on on the database server side.

This widget could not be displayed.

The machine wasn't rebooted, and the home dir appears to be intact. I believe it was a hack.

There was a phpMyAdmin instance running on the machine which was secured with HTTP basic auth.  I was the only person that would have been accessing this in the last few days, and there's quite a bit of activity from Netherlands IP addresses, with requests for the jiradb as well as others.  These requests were definitely not from me - looks like they got in via some kind of JS exploit.

Looks like a few other tables were wiped, and one or two were exported. Nothing with sensitive info - the DB was just a dev one.

As I say the data wasn't critical, but (as more of a personal exercise) I've managed to recover the data that was most useful to me - the issues - via low level recovery (thanks to undrop-for-innodb).

The VM will be going offline soon and naturally the MySQL server is off the net now.

Thanks for the help!

 

 

 

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted Tuesday in Jira

What modern development practices are at the heart of how your team delivers software?

Hey Community mates! Claire here from the Software Product Marketing team. We all know software development changes rapidly, and it's often tough to keep up. But from our research, we've found the h...

303 views 1 4
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you