Entire JIRA MySQL database suddenly gone... no cause known.

[Adam McCann]

Hello,

One day I'm using JIRA, the next I get database not found errors.  I checked and it appears my database has vanished. Entirely, and suddenly.  I don't have backups because I actually thought I installed a plugin to do that - turns out I didn't (at least on this installation).  The data isn't absolutely vital by any means, but it's certainly a pain in the ass to lose it.

I'm curious as to how it's possible the database has vanished.

Setup: MySQL server running locally with a db called 'jiradb'.  The MySQL Server was still running without any apparent problems, just the JIRA database (and corresponding MYI etc. files) are not there. I actually can't find any trace of them.

You can see how sudden it was from the logs. I've shared them on Dropbox here:

https://www.dropbox.com/sh/15a4ry6v2rzj6hn/AACGXyKkEyMaeOBuXHMF4aOqa?dl=0

If you check atlassian-jira.log, and find the first occurrence of "2016-04-08 15:56:12,190" you will see the point at which the DB was gone.

The security log (also in the shared folder, atlassian-jira-security.log) looks to shows some activity from Russian and Chinese IPs that I do not recognise (118.193.12.194, 130.193.51.39, 141.8.132.59)

Is it possible someone has got in and simply deleted my database in such a clean way?

I expect my data is long gone at this point, but any help would be greatly appreciated.

*Update*: Just used a recovery tool to check the MySQL dictionary file, and indeed I'm not going mad - the database was there with what's look like a ton of tables.  So something has entirely and cleanly wiped it!  This is extremely odd...

[Adam McCann]

The machine wasn't rebooted, and the home dir appears to be intact. I believe it was a hack.

There was a phpMyAdmin instance running on the machine which was secured with HTTP basic auth.  I was the only person that would have been accessing this in the last few days, and there's quite a bit of activity from Netherlands IP addresses, with requests for the jiradb as well as others.  These requests were definitely not from me - looks like they got in via some kind of JS exploit.

Looks like a few other tables were wiped, and one or two were exported. Nothing with sensitive info - the DB was just a dev one.

As I say the data wasn't critical, but (as more of a personal exercise) I've managed to recover the data that was most useful to me - the issues - via low level recovery (thanks to undrop-for-innodb).

The VM will be going offline soon and naturally the MySQL server is off the net now.

Thanks for the help!

[Nic Brough -Adaptavist-]

Most versions of JIRA will say "hey, there's stuff in here" if they go through a setup cycle when connected to an existing database.  If that stuff looks Jira-like, they'd ask you if you're sure you want to kill it

If you did not go through a setup process when this happened, then you've either changed the database connection settings to a database that is not a JIRA one (or doesn't exist), or you have deleted the database.  It's not a JIRA problem really, as JIRA won't do that.  Something has moved or deleted your database on on the database server side.

[Daniel Wester]

It can happen depending on the is hosted on and where you place the home dir. If you make the home dir in /tmp on a vanilla Ubuntu installation it usually gets wiped at reboot. When JIRA comes up it will then ask to be set back up again. I can't remember the behavior for JIRA7/6 but for older versions it would more than happily help you erase the database as you're setting up the instance again.

[Steven F Behnke]

I've just never heard of anything even remotely close to this happening.

You mentioned in your edit, you see tables now. Is it not possible to connect to, or can you verify you don't see custom fields, issue data, the like?

[Daniel Wester]

Where is your JIRA home directory? If it gets wiped out and JIRA is restarted you could see this.

[Adam McCann]

I wasn't doing anything with JIRA.  I just attempted to load a page in a new session and got errors, which led me to this.

The database was only secured via the JIRA login.

As mentioned, no recent backups, it's a non-AWS VM with a self-managed MySQL server - no way to get the DB.  I'm trying low level recovery tools now.

I'm more interested in knowing how this could be possible really. We only had error logging on the MySQL DB (which shows nothing), so looks like I'll be out of luck knowing what happened then.

[Steven F Behnke]

Without MySQL logs you really aren't going to get any answer.

[Steven F Behnke]

Were you upgrading JIRA or plugins? How was your database secured? Did you have a backup policy in place of any kind? Are you on AWS and using a snapshot or backup policy? Perhaps your provider has it's own back up system? Do you have any MySQL logging to provide?