JIRA had problems starting up

[Yvo]

Hello Everyone,

This morning our self-hosted JIRA installation broke down, rejecting all access requests from all users.

Since there is nothing else than "An unexpected error has occurred during startup.: We couldn't start JIRA", what should i do?

The logs: Jira logs on WeTransfer

[Nic Brough -Adaptavist- · Accepted solution]

I can't see the logs.  What do they see at the point of crash?

[Yvo]

you can't see the we-transfer link?

[Yvo]

This might help from the catalina log:

11-Dec-2018 13:04:36.111 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [webwork.action.ActionContext$1] (value [webwork.action.ActionContext$1@cb1060]) and a value of type [webwork.action.ActionContext] (value [webwork.action.ActionContext@6ccd7a7e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

There are also a whole lot GC (Allocation Failure) [PSYoungGen: messages in the atlassian-jira log file present.

[Nic Brough -Adaptavist-]

That's part of a shutdown, after the system has failed.

What does the application log say the problem is?  (Jira home/log/atlassian-jira.log)

[Yvo]

from the log:

2018-12-11 13:49:12,463 http-nio-8080-exec-12 ERROR   nr3dk7   [c.a.j.web.servlet.InternalServerErrorServlet] {errorId=f7850379-79dd-4d2f-9cfe-c3727aebc353, interpretedMsg=, cause=com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericEntityException: while updating: [GenericEntity:UserAttribute][name,login.lastLoginMillis][directoryId,1][id,10123][lowerValue,1544532552385][userId,10102][value,1544532552385] (SQL Exception while executing the following:UPDATE jiraschema.cwd_user_attributes SET user_id=?, directory_id=?, attribute_name=?, attribute_value=?, lower_attribute_value=? WHERE ID=? (The transaction log for database 'JiraDB01' is full due to 'LOG_BACKUP'.)), stacktrace=com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericEntityException: while updating: [GenericEntity:UserAttribute][name,login.lastLoginMillis][directoryId,1][id,10123][lowerValue,1544532552385][userId,10102][value,1544532552385] (SQL Exception while executing the following:UPDATE jiraschema.cwd_user_attributes SET user_id=?, directory_id=?, attribute_name=?, attribute_value=?, lower_attribute_value=? WHERE ID=? (The transaction log for database 'JiraDB01' is full due to 'LOG_BACKUP'.))

The problem is solved. There were several issues:

- Another database consumed all the IOPS

- The log of that DB caused the disk to fill up

- The SQL log of Jira was about 150 GB, shrinking had no effect

- Restarting Jira over and over caused the SQL database connection limit of 20 to be reached

The SQL server will get a restart tonight, we're hoping that the log will be written into the database at startup.

Thanks for your time.

[Pedro Felgueiras]

Don get me wrong, but ...

The logs doesn't have any helping message !!! 

[Yvo]

We've restarted the service indeed. Didn't try to restart the whole server since a bunch of other applications are running on it as well... I'll discuss it with my IT colleague.