Confluence becomes unresponsive too many open files

[Tyler Johnson]

We're having an issue with our locally hosted implementation of Confluence. The last three weekends on Saturday afternoon the catalina.out starts getting blasted with the following errors: 

11-Apr-2020 20:27:21.247 SEVERE [https-jsse-nio-8443-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:461)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)
java.util.logging.ErrorManager: 0: Unable to delete log files older than [90] days
11-Apr-2020 20:27:21.298 SEVERE [https-jsse-nio-8443-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:461)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)

We had a scheduled task for application backups enabled, I saw the recommendation was to disable that which I did since we run nightly AWS snapshots on the application server as well as the database. Disabling these backups did not resolve the issue, when this error occurs the front end becomes completely unresponsive and times out. Once the application is bounced it comes right back and is good for another week. 

Confluence specific content of /etc/limits.conf: 

# End of file
confluence soft nofile 32768
confluence hard nofile 32768

[Daniel Eads]

Hey Tyler,

Depending on which distribution you're using and if Confluence is running as a systemd service, you might need to take some additional steps. It's all outlined in this KB article - too many open files error - but I'm happy to make a smaller checklist for you based on your specific distro if you'd like. Particularly, you might just need to check:

• If you're on a debian-based distro (Ubuntu for example) - /etc/pam.d/common-session needs to have the following line:
  

session required pam_limits.so

• systemd service configurations need to have the LimitNOFILE parameter:
  

LimitNOFILE=32768

Cheers,
Daniel

[Tyler Johnson]

Hi Daniel, 

I'm not running a Debian distro, it's running on RHEL 7. I don't have a unit file enabled for systemd, when it comes up it just links to /etc/rc.d/init.d/confluence which then runs the scripts from the  /opt/atlassian/confluence/bin directory to start or stop confluence. Do you think switching to a unit file with the LimitNOFILE parameter would resolve my issue?