Site went down

[Kurasi Vishnu Priya]

Hi

We planned to upgrade our jira from 8.4 to 8.7 but before upgrade we are seeing the db connections went down and jira stage site has complete;y down,

We are not sure why site went down , Now recently the jira has started to reboot very frequently while trying to access in server.

we are seeing error as 

Broadcast message from root user

The system  is going to power off now

Kindly help on this, We need our site to be up immediately

Thanks

Vishnu Priya K

[Charlie Misonne · Accepted solution]

The system  is going to power off now

This definitely meens your server was purposefully stopped by someone or something (a script?).
You can check last shutdowns with:

last -x | less

Perhaps you'll also find some useful information in /var/log/messages

If the server running Jira is stopped of course the DB connections will go down as well.

Now recently the jira has started to reboot very frequently while trying to access in server.

Does this happen on regular intervals or fixed moments? In that case there could be a scheduled shutdown of your server.

Either way: you should find out who or what is stopping your server.

[Kurasi Vishnu Priya]

Right now the jira server is stable but we are not sure why our site is down, can u plz guide in that.

Is there any possible way to have a screen share or call in community support if so let me know on it

Thanks

Priya K

[Kurasi Vishnu Priya]

may i know where dns will be placed in jira server

[Charlie Misonne]

I'm afraid the Community will not organize screen share sessions. It's also not the best place for urgent incidents. Atlassian has a support channel for that: https://support.atlassian.com/contact/
But they will only support you in Atlassian related matters. And it seems like your systems was shut down for other reasons.

If you want to troubleshoot why your site is down:
Start you Jira application and check your log file while Jira tries to start. You should get errors if anything goes wrong.

What do you mean with: "may i know where dns will be placed in jira server"?
Jira uses the DNS from your server. There is no separate DNS setting in the application

[Nic Brough -Adaptavist-]

Just to be clear - the problem here is not Jira.

"Broadcast message from root user: The system  is going to power off now" is a message sent to all active users from the user who is shutting down or restarting the system.  It's nothing to do with Jira, one of your *nix system admins is deliberately stopping/restarting the server with standard operating system commands.

[Kurasi Vishnu Priya]

My Jira is not at all getting started in linux server , Its showing some kind of error like this

2021-07-08 06:46:36,514 JIRA-Bootstrap ERROR      [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ]

Can you help me on this , 

[Kurasi Vishnu Priya]

What need to configure to become my jira website up

[Charlie Misonne]

Is this the only error you get? I'd expect more errors higher in the log files. Can you check that?

[Kurasi Vishnu Priya]

2021-07-08 06:46:36,186 JIRA-Bootstrap ERROR [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.plugin.osgi.container.OsgiContainerException: Unable to clean the cache directory: /var/atlassian/application-data/efs/plugins/.osgi-plugins/felix
at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.validate(DefaultOsgiPersistentCache.java:71)
at com.atlassian.plugin.osgi.container.impl.DefaultOsgiPersistentCache.<init>(DefaultOsgiPersistentCache.java:38)
at com.atlassian.jira.plugin.PluginPath$JiraHomeAdapter.<init>(PluginPath.java:135)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
... 1 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.picocontainer.injectors.AbstractInjector.newInstance(AbstractInjector.java:145)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:342)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:631)
at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:40)
at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:646)
at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:677)
at com.atlassian.jira.component.pico.CachingMutablePicoContainer.getComponent(CachingMutablePicoContainer.java:139)
at com.atlassian.jira.component.pico.ComponentContainer.getComponent(ComponentContainer.java:555)
at com.atlassian.jira.component.pico.ComponentManager.lambda$maybeGetComponent$3(ComponentManager.java:474)
at java.util.Optional.map(Optional.java:215)
at com.atlassian.jira.component.pico.ComponentManager.maybeGetComponent(ComponentManager.java:474)
at com.atlassian.jira.component.pico.ComponentManager.getComponent(ComponentManager.java:462)
at com.atlassian.jira.component.pico.ComponentManager.loadPluginSystemEventListeners(ComponentManager.java:277)
at com.atlassian.jira.component.pico.ComponentManager.earlyStartPluginSystem(ComponentManager.java:237)
at com.atlassian.jira.upgrade.PluginSystemLauncher.start(PluginSystemLauncher.java:45)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:301)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:196)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:104)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:102)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to list contents of /var/atlassian/application-data/efs/plugins/.osgi-plugins/felix/felix-cache
at org.apache.commons.io.FileUtils.verifiedListFiles(FileUtils.java:1751)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1716)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
... 77 more
2021-07-08 06:46:36,514 JIRA-Bootstrap ERROR [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ]

These only showing up in log files , not sure why jira has locked

When we try to start jira it showing like,

[jira@jira bin]$ sh start-jira.sh

To run JIRA in the foreground, start the server with start-jira.sh -fg
executing as current user

`sMMMMMMMMMMMMMM+
MMMMMMMMMMMMMM
:sdMMMMMMMMMMM
MMMMMM
`sMMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sMMMMMMMMMMM MMMMM
MMMMMM `UOJ
`sMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sdMMMMMMMMMM MMMMM
MMMMMM `UOJ
MMMMMM
+MMMMM
MMMMM
`UOJ

Atlassian Jira
Version : 8.4.0

If you encounter issues starting or stopping JIRA, please see the Troubleshooting guide at https://docs.atlassian.com/jira/jadm-docs-084/Troubleshooting+installation

Server startup logs are located in /opt/atlassian/jira/logs/catalina.out
Using CATALINA_BASE: /opt/atlassian/jira
Using CATALINA_HOME: /opt/atlassian/jira
Using CATALINA_TMPDIR: /opt/atlassian/jira/temp
Using JRE_HOME: /opt/atlassian/jira/jre/
Using CLASSPATH: /opt/atlassian/jira/bin/bootstrap.jar:/opt/atlassian/jira/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/atlassian/jira/work/catalina.pid
Existing PID file found during start.
Tomcat appears to still be running with PID 6745. Start aborted.
If the following process is not a Tomcat process, remove the PID file and try again:
UID PID PPID C STIME TTY TIME CMD
jira 6745 1 0 Jul08 ? 00:02:15 /opt/atlassian/jira/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.uti

Kindly help its an urgent req

[Nic Brough -Adaptavist-]

I suspect what has happened here is a compounded problem.  

I think you've tried to start Jira up as the wrong user, then one of your system admins halted the system, which forcibly killed the process.  You've then tried to start it again as the right user, but because the other user has taken ownership of some of the files, it can't write what it needs to (clearing the cache for example, and removing the existing "Jira is running" lock files).

I would check to see if Jira is partially running, and shut it down if is is, then check all the ownership and permissions on the Jira installation and home (data) directories, making sure the Jira user has full access (usually just making sure they own everything throughout those two directories) and restart it as the right user.