JIRA, Confluence, Crowd, Stash(bitbucker) don't boot up.

[Krzysztof Macewicz]

I need some help with a very annoying problem.

My company uses the Atlassian stack: JIRA, Bitbucker, Confluence and Crowd. We got all of them in one remote server. To log into the applications we use Nginx.

So the problem is:

Often (one time per two days) all the atlassian applications stop working. It's really strange, when I try to log into JIRA it says 'Server internal error', on other applications (BB, Crowd and Conf) I got an error that Nginx didn't find any applications under proxy. I'm really confused about this issue. For now I solve it by restarting all applications (there are scripts inside the application to restart it (stop and start)), then I reboot the PC and restart Nginx. Sometimes the applications are up after first reboot, sometimes after third or even fourth reboot. It's very annoying. Nginx is not a problem, because other applications like nexus are working. On my remote server I use ubuntu 16.04.

My questions are:

- Is there any procedure to check health of the applications?
- Why do these applications stop working and crash so often?
- Is there any good practice to manage this problem?

Cheers

[Marcin Kempa]

Hi @Krzysztof Macewicz,

Could you post some of your logs when you experience downtime? That would help us to understand the problem.

Cheers,

Marcin Kempa

[Krzysztof Macewicz]

Hi Marcin,

Yes, of course. Actually today morning it stopped working so the log is fresh. Yesterday it was working till 7 pm. It was the time when last person left office. My colleges told me that JIRA is not working since 9 AM.

I sent you screen when i try to connect jira.

Can you please give me e-mail to you? Log has 126,6 kB.

Cheers,

Krzysztof

[Marcin Kempa]

Sure, my email is: mkempa@atlassian.com

[Krzysztof Macewicz]

Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem                                                   Size    Used Avail    Use% Mounted on
/dev/mapper/vg_atlassian-lv_root    64G    59G   2,2G     97%   /
tmpfs                                                              11G    0         11G      0%     /dev/shm
/dev/sda1                                                     485M 34M   426M  8%     /boot
/dev/mapper/vg_atlassian-lv_home 45G    9,9G   33G     24% /home

Memory (watch -n 5 free -m)

                             total    used    free     shared buffers cached
Mem:                 22007 12899 9107   0            124        5939
-/+ buffers/cache:        6836  15171
Swap:                 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

[Lars Olav Velle]

Is your nexus service is running as a different user?

My guess is that the problem is too many open files.

https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

-Lars

[Krzysztof Macewicz]

Thanks Lars for suggestion.

I checked max files that can be open, result:

cat /proc/sys/fs/file-max
2237374

Should i increase it?

My Nexus is running on diffrent virtual machine.

[Lars Olav Velle]

The /proc/sys/fs/file-max would be the maximum allowed for the whole server. Leave that as is.

Here is output from a server with increased maximum  open files for the user otherUser:

[root@server ~]# ulimit -Hn
4096
[root@server ~]# ulimit -Sn
1024

[root@server ~]# su otherUser

[otherUser@server]$ ulimit -Sn
131072
[otherUser@server]$ ulimit -Hn
131072

These settings you`ll find in:

/etc/security/limits.conf

otherUser hard nofile 131072
otherUser soft nproc 131072
otherUser hard nproc 131072
otherUser soft nofile 131072
# End of file

[Krzysztof Macewicz]

Thanks for reply

I run as root.

[root@Atlassian ~]# ulimit -Hn
4096
[root@Atlassian ~]# ulimit -Sn
1024

Is it correct?

Open: /etc/security/limits.conf

add: root soft nofile 4096
     root hard nofile 10240

Cheers,

Krzysztof

[Lars Olav Velle]

Actually I am not entirely sure as I have never run any applications as root.

If you want to determine if this is the actual problem you might want to test with unlimited first. 

The following will set unlimited, but will be back to standard after a reboot.

(su - username)

ulimit

[Krzysztof Macewicz]

You're right. I haven't configure this machine.

There is a user jira and crowd.

Both got

ulimit -Hn
4096

ulimit -Sn
1024

To make this users unlimit, I should:

-log into root user

-open /etc/security/limits.conf

-add: jira soft nofile unlimited

    jira hard nofile unlimited

same for crowd user.

Am I correct?

Thanks a lot

Krzysztof

[Christian Glockner]

Hi Krzsysztof,

Can you give more details about your system, such as how much RAM you have on the machine?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Krzysztof Macewicz]

Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem                                                   Size    Used Avail    Use% Mounted on
/dev/mapper/vg_atlassian-lv_root    64G    59G   2,2G     97%   /
tmpfs                                                              11G    0         11G      0%     /dev/shm
/dev/sda1                                                     485M 34M   426M  8%     /boot
/dev/mapper/vg_atlassian-lv_home 45G    9,9G   33G     24% /home

Memory (watch -n 5 free -m)

                             total    used    free     shared buffers cached
Mem:                 22007 12899 9107   0            124        5939
-/+ buffers/cache:        6836  15171
Swap:                 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)