JIRA 7.04 SSL File Location/Tomcat KeysStore File and LDAP USer Directory Access Issues

[roozadhami]

We have a server for which we are running JIRA 7.04 using start/stop script (not as a service) and that we also use LDAP directory service in order to allow our client users to log onto JIRA dashboard for which they use JIRA to submit forms. Currently we have lots of users that are not able to access the JIRA Service Desk URL/System dashboard and that the get LDAP errors like of  "Error occurred while trying to authenticate user 'jared.noel@vnsny.org')" as shown below:

2019-05-23 11:20:42,239 http-nio-8080-exec-39 ERROR anonymous 680x11937x1 oec13g 65.51.234.2,10.154.71.241 /servicedesk/customer/user/login [c.a.c.manager.application.ApplicationServiceGeneric] Directory 'VNSNY AD' is not functional during authentication of 'jared.noel@vnsny.org'. Skipped.
2019-05-23 11:20:42,239 http-nio-8080-exec-39 ERROR anonymous 680x11937x1 oec13g 65.51.234.2,10.154.71.241 /servicedesk/customer/user/login [c.a.j.security.login.JiraSeraphAuthenticator] Error occurred while trying to authenticate user 'jared.noel@vnsny.org'.
com.atlassian.crowd.exception.runtime.OperationFailedException

---
019-05-23 11:20:33,789 http-nio-8080-exec-26 ERROR anonymous 680x11928x1 1cv40f4 65.51.234.100,10.154.71.241 /servicedesk/customer/user/login [c.a.c.manager.application.ApplicationServiceGeneric] Directory 'VNSNY AD' is not functional during authentication of '59440@vnsny.org'. Skipped.
2019-05-23 11:20:33,790 http-nio-8080-exec-26 ERROR anonymous 680x11928x1 1cv40f4 65.51.234.100,10.154.71.241 /servicedesk/customer/user/login [c.a.j.security.login.JiraSeraphAuthenticator] Error occurred while trying to authenticate user '59440@vnsny.org'.

2019-05-23 19:41:25,903 Caesium-1-2 INFO ServiceRunner [c.a.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete for directory [ 10200 ] in [ 175ms ]
2019-05-23 19:41:25,940 Caesium-1-2 ERROR ServiceRunner [c.atlassian.scheduler.JobRunnerResponse] Unable to synchronise directory

We have tried to tried to modify some of the options for the affected User Directory (i.e. VNSNY AD) in JIRA Administration/User Directory control panel but have not been successful in getting the issue resolved. the hostname for that affected User Directory of "VNSNY AD" is adldap.vnsny.org which is a VM/system that is on the client's side/network. when we do a dig command for that VM from the JIRA server it shows that the SOA authority for that server (i.e. ) is ns-217.awsdns-27.com, see below:

# dig adldap.vnsny.org

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7 <<>> adldap.vnsny.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7713
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;adldap.vnsny.org. IN A

;; AUTHORITY SECTION:
vnsny.org. 1800 IN SOA ns-217.awsdns-27.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 7 msec
;; SERVER: 10.255.255.11#53(10.255.255.11)

We have thought that the SSL certificate for the JIRA might have been a culprit for this problem and that since JIRA server is also incorporating a Tomcat server for its operation and since the Tomcat uses JAVA Key-store file for storing its certificates therefore, we need to know where exactly that keystore file is located (our Jira base installation directory is '/apps/atlassian-install' ) and how we can make sure that the installed certificate file is NOT expired.   Also another question is if the LDAP server as configured on the User directory control panel, i.e. adldap.vnsny.org should get/receive its SOA information from Amazon (i.e . ns-217.awsdns-27.com. awsdns-hostmaster.amazon.com)?   Please treat this issue with a very high urgency as this issue is affecting a large number of the JIRA Service Desk application users.  Please feel free to contact me on my Cell# 512-228-7042 or via my Email as :Roozbeh.Adhami@ipsfot.com. We really really in a tight situation.  

Best Regards,
RRoozbeh Adhami
Senior Application Support Engineer

IPsoft Inc.
 

[Andy Heinzer · Accepted solution]

Hi,

The most telling error you have posted so far is

2019-05-23 19:41:25,903 Caesium-1-2 INFO ServiceRunner [c.a.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete for directory [ 10200 ] in [ 175ms ]

It tells us that Jira has not been able to sync this user directory at the very least.  In turn, Jira is likely unable to communicate with the directory, or doesn't have the exact settings it needs to be able to communicate with this directory.  It could be something like an SSL certificate problem, but with the information we have right now, I'm not ready to jump to that conclusion just yet.   The only reason Jira would need an SSL cert to connect to an LDAP server is if that LDAP was running in SSL/TLS, Jira was configured to use that secured address.  In that case, yes, a cert will have to exist in the truststore that Jira is using.  Where is that truststore?  Well that depends on which instance of Java Jira is starting up with.  If you defined the $JAVA_HOME variable in this system then that certificate would by default be in $JAVA_HOME/jre/lib/security/cacerts

However not all Jira admins do this.  In cases where you might be running a bundled JRE with Jira (if you used the bin or exe install package), the location tends to be the $JIRA_Install/jre/lib/security/cacerts

If this is the case here, then it might help to try to walk back through the steps in Connecting to SSL services.  It explains the steps and tools you can use to add an SSL cert to Jira.  Perhaps you could use the portecle app mentioned here to examine the certificates in the trust store here more closely.

However I would still like to know more about your setup here.  The errors provided so far don't tell us definitively the source of the problem, so far they are just the symptoms of the problem.   I would be interested to see if we can take a closer look at the $JIRAHOME/log/atlassian-jira.log file at the time that Jira attempts to perform this sync.   Perhaps you can generate a support zip, this will have a considerable amount of more environmental details and logs from your site that could help us to better understand the source of the problem here.

Andy

[roozadhami]

Hello Andrew, 

Thanks for your reply email.  The issue has been resolved. The problem with the LDAP was with the permissions on the  name-server's configuration file on the server (i.e. resolv.conf).   As the issue was resulted with the change of permissions on the resolv.conf file to be 0600 (only readable/writable by root user) instead of being 0644. Have you ever encounter any situation/cases where JIRA software might have caused the permission change on system file such as resolv.conf?  I know that JIRA application runs under jira user and therefore, I am not completely sure if the file was changed by JIRA and/or its software.  Any information that you might have will be appreciated.

Regards,

Rooz Adhami

IPsoft Inc.

[Andy Heinzer]

In my experience, No, I have never seen Jira itself be able to make changes to a resolv.conf file.  When you connect Jira to an external user directory, you have to define an LDAP admin account credentials within Jira.  If you configure Jira to use the Read/Write option for that directory it is possible that Jira could make changes to the LDAP directory, but these would consist of user account changes like usernames, password, or potentially things like groups and memberships.  I can't see how changes to that aspect could directly or indirectly make changes to a resolv.conf file here.

Even for the user account running the Jira / java process itself, I have not see that account ever try to modify such a file.

Glad to hear things sound like they are working again here.

Andy