I need some help with a very annoying problem.
My company uses the Atlassian stack: JIRA, Bitbucker, Confluence and Crowd. We got all of them in one remote server. To log into the applications we use Nginx.
So the problem is:
Often (one time per two days) all the atlassian applications stop working. It's really strange, when I try to log into JIRA it says 'Server internal error', on other applications (BB, Crowd and Conf) I got an error that Nginx didn't find any applications under proxy. I'm really confused about this issue. For now I solve it by restarting all applications (there are scripts inside the application to restart it (stop and start)), then I reboot the PC and restart Nginx. Sometimes the applications are up after first reboot, sometimes after third or even fourth reboot. It's very annoying. Nginx is not a problem, because other applications like nexus are working. On my remote server I use ubuntu 16.04.
My questions are:
- Is there any procedure to check health of the applications?
- Why do these applications stop working and crash so often?
- Is there any good practice to manage this problem?
Cheers
Could you post some of your logs when you experience downtime? That would help us to understand the problem.
Cheers,
Marcin Kempa
Hi Marcin,
Yes, of course. Actually today morning it stopped working so the log is fresh. Yesterday it was working till 7 pm. It was the time when last person left office. My colleges told me that JIRA is not working since 9 AM.
I sent you screen when i try to connect jira.
Can you please give me e-mail to you? Log has 126,6 kB.
Cheers,
Krzysztof
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Sure, my email is: mkempa@atlassian.com
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Christian,
Yes, of course.
disk usage (df -h):
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_atlassian-lv_root 64G 59G 2,2G 97% /
tmpfs 11G 0 11G 0% /dev/shm
/dev/sda1 485M 34M 426M 8% /boot
/dev/mapper/vg_atlassian-lv_home 45G 9,9G 33G 24% /home
Memory (watch -n 5 free -m)
total used free shared buffers cached
Mem: 22007 12899 9107 0 124 5939
-/+ buffers/cache: 6836 15171
Swap: 3967 0 3967
The system looks like is CentOS, sorry for mistake.
cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Is your nexus service is running as a different user?
My guess is that the problem is too many open files.
https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
-Lars
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks Lars for suggestion.
I checked max files that can be open, result:
cat /proc/sys/fs/file-max
2237374
Should i increase it?
My Nexus is running on diffrent virtual machine.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
The /proc/sys/fs/file-max would be the maximum allowed for the whole server. Leave that as is.
Here is output from a server with increased maximum open files for the user otherUser:
[root@server ~]# ulimit -Hn
4096
[root@server ~]# ulimit -Sn
1024
[root@server ~]# su otherUser
[otherUser@server]$ ulimit -Sn
131072
[otherUser@server]$ ulimit -Hn
131072
These settings you`ll find in:
/etc/security/limits.conf
otherUser hard nofile 131072
otherUser soft nproc 131072
otherUser hard nproc 131072
otherUser soft nofile 131072
# End of file
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thanks for reply
I run as root.
[root@Atlassian ~]# ulimit -Hn
4096
[root@Atlassian ~]# ulimit -Sn
1024
Is it correct?
Open: /etc/security/limits.conf
add: root soft nofile 4096
root hard nofile 10240
Cheers,
Krzysztof
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Actually I am not entirely sure as I have never run any applications as root.
If you want to determine if this is the actual problem you might want to test with unlimited first.
The following will set unlimited, but will be back to standard after a reboot.
(su - username)
ulimit
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You're right. I haven't configure this machine.
There is a user jira and crowd.
Both got
ulimit -Hn
4096
ulimit -Sn
1024
To make this users unlimit, I should:
-log into root user
-open /etc/security/limits.conf
-add: jira soft nofile unlimited
jira hard nofile unlimited
same for crowd user.
Am I correct?
Thanks a lot
Krzysztof
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Krzsysztof,
Can you give more details about your system, such as how much RAM you have on the machine?
Cheers,
Christian
Premier Support Engineer
Atlassian
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Christian,
Yes, of course.
disk usage (df -h):
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_atlassian-lv_root 64G 59G 2,2G 97% /
tmpfs 11G 0 11G 0% /dev/shm
/dev/sda1 485M 34M 426M 8% /boot
/dev/mapper/vg_atlassian-lv_home 45G 9,9G 33G 24% /home
Memory (watch -n 5 free -m)
total used free shared buffers cached
Mem: 22007 12899 9107 0 124 5939
-/+ buffers/cache: 6836 15171
Swap: 3967 0 3967
The system looks like is CentOS, sorry for mistake.
cat /etc/*release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.