JIRA, Confluence, Crowd, Stash(bitbucker) don't boot up. Edited

I need some help with a very annoying problem.

My company uses the Atlassian stack: JIRA, Bitbucker, Confluence and Crowd. We got all of them in one remote server. To log into the applications we use Nginx.

So the problem is:

Often (one time per two days) all the atlassian applications stop working. It's really strange, when I try to log into JIRA it says 'Server internal error', on other applications (BB, Crowd and Conf) I got an error that Nginx didn't find any applications under proxy. I'm really confused about this issue. For now I solve it by restarting all applications (there are scripts inside the application to restart it (stop and start)), then I reboot the PC and restart Nginx. Sometimes the applications are up after first reboot, sometimes after third or even fourth reboot. It's very annoying. Nginx is not a problem, because other applications like nexus are working. On my remote server I use ubuntu 16.04.

My questions are:

- Is there any procedure to check health of the applications?
- Why do these applications stop working and crash so often?
- Is there any good practice to manage this problem?

 

Cheers

3 answers

0 votes

Hi Krzsysztof,

Can you give more details about your system, such as how much RAM you have on the machine?

Cheers,

Christian

Premier Support Engineer

Atlassian

Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem                                                   Size    Used Avail    Use% Mounted on
/dev/mapper/vg_atlassian-lv_root    64G    59G   2,2G     97%   /
tmpfs                                                              11G    0         11G      0%     /dev/shm
/dev/sda1                                                     485M 34M   426M  8%     /boot
/dev/mapper/vg_atlassian-lv_home 45G    9,9G   33G     24% /home

Memory (watch -n 5 free -m)

                             total    used    free     shared buffers cached
Mem:                 22007 12899 9107   0            124        5939
-/+ buffers/cache:        6836  15171
Swap:                 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem                                                   Size    Used Avail    Use% Mounted on
/dev/mapper/vg_atlassian-lv_root    64G    59G   2,2G     97%   /
tmpfs                                                              11G    0         11G      0%     /dev/shm
/dev/sda1                                                     485M 34M   426M  8%     /boot
/dev/mapper/vg_atlassian-lv_home 45G    9,9G   33G     24% /home

Memory (watch -n 5 free -m)

                             total    used    free     shared buffers cached
Mem:                 22007 12899 9107   0            124        5939
-/+ buffers/cache:        6836  15171
Swap:                 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

Is your nexus service is running as a different user?

 

My guess is that the problem is too many open files.

https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

 

-Lars

Thanks Lars for suggestion.

I checked max files that can be open, result:

cat /proc/sys/fs/file-max
2237374

Should i increase it?

My Nexus is running on diffrent virtual machine.

The /proc/sys/fs/file-max would be the maximum allowed for the whole server. Leave that as is.

Here is output from a server with increased maximum  open files for the user otherUser:

 

[root@server ~]# ulimit -Hn
4096
[root@server ~]# ulimit -Sn
1024

[root@server ~]# su otherUser

[otherUser@server]$ ulimit -Sn
131072
[otherUser@server]$ ulimit -Hn
131072

 

These settings you`ll find in:

/etc/security/limits.conf

otherUser hard nofile 131072
otherUser soft nproc 131072
otherUser hard nproc 131072
otherUser soft nofile 131072
# End of file

Thanks for reply

I run as root.

[root@Atlassian ~]# ulimit -Hn
4096
[root@Atlassian ~]# ulimit -Sn
1024

Is it correct?

Open: /etc/security/limits.conf

add: root soft nofile 4096
     root hard nofile 10240

Cheers,

Krzysztof

Actually I am not entirely sure as I have never run any applications as root.

If you want to determine if this is the actual problem you might want to test with unlimited first. 

The following will set unlimited, but will be back to standard after a reboot.

(su - username)

ulimit

You're right. I haven't configure this machine.

There is a user jira and crowd.

Both got

ulimit -Hn
4096

ulimit -Sn
1024

To make this users unlimit, I should:

-log into root user

-open /etc/security/limits.conf

-add: jira soft nofile unlimited

    jira hard nofile unlimited

same for crowd user.

Am I correct?

 

Thanks a lot

Krzysztof

0 votes
Marcin Kempa Atlassian Team Nov 28, 2017

Hi @Krzysztof Macewicz,

Could you post some of your logs when you experience downtime? That would help us to understand the problem.

 

Cheers,

Marcin Kempa

Hi Marcin,

Yes, of course. Actually today morning it stopped working so the log is fresh. Yesterday it was working till 7 pm. It was the time when last person left office. My colleges told me that JIRA is not working since 9 AM.

I sent you screen when i try to connect jira.Screenshot from 2017-11-28 10-39-24.png

Can you please give me e-mail to you? Log has 126,6 kB.

 

Cheers,

Krzysztof

Marcin Kempa Atlassian Team Nov 28, 2017

Sure, my email is: mkempa@atlassian.com

Suggest an answer

Log in or Join to answer

Stay in touch

Be the first to know what's trending on Atlassian Community