Confluence user can't login due to JIRA domain update

[Celtis Omaris]

The way we have set up jira and confluence servers is that the Jira instance has all the users and their credentials and confluence uses those users and credentials to login to confluence.

Our jira instance was running without SSL and on port 8080 and we have since moved it to SSL and now the users can't use their credentials to log into confluence.

I've tried restoring the domain to port 8080, logging into confluence and keeping the session logged in so I can update the application link after transitioning back to SSL but that doesn't work.

I did a grep on all the files on the server and couldn't see where it might be set up so I can change it.

I did a search in the confluence DB and found some entries that looked like they needed updating also, and I ran the below commands to update them but still didnt work.

UPDATE cwd_directory_attribute
SET attribute_value = REPLACE(attribute_value, 'http://DOMAIN:8080', 'https://DOMAIN')
WHERE attribute_value LIKE '%'http://DOMAIN:8080%'

UPDATE BANDANA
SET BANDANAVALUE = REPLACE(BANDANAVALUE, 'DOMAIN:8080', 'https://DOMAIN')
WHERE BANDANAVALUE LIKE '%DOMAIN:8080%'

Any more Ideas?

[Andy Heinzer]

Hi,

I understand that you have recently changed the site address where Jira is reachable (from http to an https address) and since that time, users are unable to login to Confluence.  I see that you have tried to use some SQL to update values here.  However I would recommend against this method to try to correct this problem.  It might work, but sometimes there are caches that have to be cleared in the application when making direct SQL changes like this that can only be achieved by restarting the application itself.

Instead, I would recommend trying to follow Restore Passwords To Recover Admin User Rights to then at least be able to have one user internal to Confluence as a admin be able to login.   From there you will probably need to setup this application link and Jira user server settings again. But before you do that, I have some questions for you:

1. Is Jira's Tomcat handing SSL directly?  Or do you have a proxy / loadbalancer in front of Jira's Tomcat that is terminating the SSL connection here?
2. Is Confluence using HTTP or HTTPS/SSL at all?

Ideally, I would prefer to try to have both applications (Jira and Confluence) using the same protocol when communicating with each other (either both using HTTP or both using HTTPS).  It tends to avoid the mixed content errors for the sake of application links, and other complications that can prevent this integration from working.    Both applications will use the $install/conf/server.xml file to manage what Connectors are setup for each respective Tomcat instance. It is possible to have more than one connector defined here, which might be needed if say Jira is using HTTPS and Confluence is not.

The additional tricky part though here is that using SSL has added requirements.  Namely, each application has to then have the SSL certificate of the other application in its Java truststore, otherwise it won't be able to establish an SSL connection here.  So in your case, at the very least we would want to make sure that Confluence's java truststore contains this new Jira SSL certificate.  Steps on how you can import this are explained in such documents as How to import an existing SSL certificate for use in Tomcat and/or How to import a public SSL certificate into a JVM.

And conversely, if Confluence is also running over HTTPS, you would want to make sure that Jira has the Confluence certificate in its truststore.

It would be helpful to take a closer look at the $ConfluenceHOME/logs/atlassian-confluence.log file as well.  I suspect this has some other details in the stacktraces that might help to clarify the connection problem here.  I am thinking that you might be encountering the problem documented in Unable to connect to SSL services due to "PKIX Path Building Failed" error.  But even if you are not, the diagnosis steps in that article could still be helpful to follow here just so we can learn more about your specific environment.

Andy

[Celtis Omaris]

Hi Andy,

Thanks for your helpful response to answer your questions

1. Jira is running behind a proxy, the way I have it set us is via an AWS instance that is called via a LoadBalancer running on SSL 443 and routing directly to port 8080 on the instance - This is the same set up I have for the confluence server but routing to port 8090
2. Yes it is behind SSL as described in 1. and it has been working like that while jira was still on public port 8080, when I changed jira confluence failed to login as described

Because I'm using AWS's SSL certificates I don't believe I have access to export them and import them as you mentioned.

I will also paste the stack trace from that log you mentioned in the following post, you might be able to pinpoint the issue from that, I hope.

Thank you again for your thorough reply

Omar

[Celtis Omaris]

2019-09-10 11:05:09,399 ERROR [http-nio-8090-exec-20] [[Standalone].[localhost].[/].[noop]] log Servlet.service() for servlet [noop] in context with path [] threw exception
java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:140)
        at com.atlassian.plugin.cache.filecache.impl.OneStreamCache.stream(OneStreamCache.java:65)
        at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.cache(FileCacheImpl.java:100)
        at com.atlassian.plugin.webresource.impl.http.Controller.sendCachedInProduction(Controller.java:355)
        at com.atlassian.plugin.webresource.impl.http.Controller.sendCached(Controller.java:305)
        at com.atlassian.plugin.webresource.impl.http.Controller.serveResources(Controller.java:234)
        at com.atlassian.plugin.webresource.impl.http.Controller.serveBatch(Controller.java:111)
        at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:92)
        at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:83)
        at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.callHandler(BaseRouter.java:197)
        at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.dispatch(BaseRouter.java:163)
        at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:65)
        at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:28)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        ...
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:758)
        at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:663)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:368)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:346)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
        at com.atlassian.confluence.web.filter.ResponseOutputStreamFilter$NoopAfterCloseOutputStream.write(ResponseOutputStreamFilter.java:100)
        at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
        at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
        at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
        at com.atlassian.gzipfilter.GzipResponseStream.write(GzipResponseStream.java:46)
        at com.atlassian.gzipfilter.RoutableServletOutputStream.write(RoutableServletOutputStream.java:134)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
        at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:135)
        ... 347 more
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1223)
        at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:743)
        at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:513)
        at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:451)
        at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:530)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:112)
        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:189)
        at org.apache.coyote.Response.doWrite(Response.java:599)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:328)
        ... 362 more
2019-09-10 11:05:14,699 WARN [http-nio-8090-exec-8] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$7 Failed to read entry from cache 'com.atlassian.confluence.cache.jcaptcha.ConfluenceCachingCaptchaStore': Failed due to UNCLASSIFIED_FAILURE
 -- referer: https://<confluence-site>/login.action?os_destination=%2F | url: /dologin.action | traceId: 57918e23ccaf78ee
2019-09-10 11:05:14,700 INFO [http-nio-8090-exec-8] [confluence.security.login.DefaultLoginManager] recordLoginFailure
Failed login attempt for user 'omar':

[Celtis Omaris]

I also ran SSLPoke as suggested here

https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html

And it successfully connected

[Celtis Omaris]

I logged in as recovery_user as suggested in the First link you sent and tried to change my admin user's password and got the below error

2019-09-10 11:42:52,005 DEBUG [http-nio-8090-exec-2] [integration.rest.service.RestExecutor] buildUrl Constructed https://<jira-site>/rest/usermanagement/1/user/password?username=omar
2019-09-10 11:42:53,098 DEBUG [http-nio-8090-exec-2] [integration.rest.service.RestExecutor] executeCrowdServiceMethod Cache response for PUT https://<jira-site>/rest/usermanagement/1/user/password?username=omar was CACHE_MISS
2019-09-10 11:42:53,353 WARN [http-nio-8090-exec-2] [confluence.user.actions.SetUserPasswordAction] doSet Failed to update user password. Cause: ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. client.forbidden.exception Go to Jira home
-- referer: https://<confluence-site>/admin/users/setuserpassword.action?username=omar | url: /admin/users/dosetuserpassword.action | traceId: 4bb23f445f8aed1c | userName: recovery_admin | action: dosetuserpassword
2019-09-10 11:42:53,356 WARN [http-nio-8090-exec-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #1780071959)
-- referer: https://<confluence-site>/admin/users/setuserpassword.action?username=omar | url: /admin/users/dosetuserpassword.action | traceId: 4bb23f445f8aed1c | userName: recovery_admin | action: dosetuserpassword

[Andy Heinzer]

Hi Omar,

I am not surprised that trying to change the password failed here.  That recovery admin account is there to make sure you can login as an admin to Confluence first.  That will be necessary in order to make the latter changes in regards to the user server. 

The fact that you are getting a 403 error when trying to change the password is just showing us that there is a problem with Confluence being unable to authenticate to Jira at the address expected.   Which is the same problem as to why users can't login to Confluence in the first place, either Jira, Confluence, or both see an error in this connection.

Which is why I would recommend walking back through the steps in Connecting to Crowd or Jira for User Management.  This way you can try to setup this integration once more afresh.  I believe it will be easier to remove the existing settings and create a new entry on both sides.