if you have backup of HOME dir before MySQL upgrade than it could not be hard to recover it.
You can follow up article for migration to another server (it's like your situation).
Or as first you can try install MySQL 5.7 and recover data into new DB from backup. Than you just need edit config in HOMEDIR/dbconfig.xml. If this not help then follow linked article.
Hope it helps and you will recover it in short time.
In future - be careful with upgrades on Jira server. Everytime check Supported platforms in Jira documentation - like this https://confluence.atlassian.com/adminjiraserver073/supported-platforms-861253018.html
Important Change; InnoDB: The following
InnoDB file format configuration options were deprecated in MySQL 5.7.7 and are now removed:
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events