Many of our users cannot access Stash at the moment. The error says, "The remote authentication server is not available. Please try again later."

[mcc24mm1]

Many of our users cannot access Stash at the moment. We suspect that the issue is related to JIRA groups because we recently made changes to allow external users to be in the system and only see one project. 

What could be causing this error??

[pipergragg]

I also work for Gramercy Tech. Here is the log that we are getting. 

3201 2015-11-16 17:55:52,138 ERROR [http-nio-7990-exec-7] @1C8BAPVx1075x32831x0 1h57xk4 67.244.86.216,127.0.0.1 "POST /j_stash_security_check HTTP/1.0" c.a.c.m.a.ApplicationServiceGeneric Directory 'JIRA User Directory' is not functional during authentication of 'quadr   3201 os'. Skipped.

   3202 2015-11-16 17:55:52,154 WARN  [http-nio-7990-exec-7] @1C8BAPVx1075x32831x0 1h57xk4 67.244.86.216,127.0.0.1 "POST /j_stash_security_check HTTP/1.0" c.a.s.i.s.s.PluginAuthenticationProvider Could not authenticate quadros; authentication by com.atlassian.stash.stash-   3202 authentication:crowdHttpAuthHandler failed

   3203 com.atlassian.stash.user.AuthenticationSystemException: The remote authentication server is not available. Please try again later.

   3204         at com.atlassian.stash.internal.crowd.RiotPolice.authenticate(RiotPolice.java:113) ~[stash-service-impl-3.8.0.jar:na]

   3205         at com.atlassian.stash.internal.user.DefaultUserService.authenticate(DefaultUserService.java:107) ~[stash-service-impl-3.8.0.jar:na]

   3206         at com.atlassian.stash.internal.auth.EmbeddedCrowdHttpAuthenticationHandler.authenticate(EmbeddedCrowdHttpAuthenticationHandler.java:40) ~[stash-service-impl-3.8.0.jar:na]

   3207         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider$1.perform(PluginAuthenticationProvider.java:97) ~[PluginAuthenticationProvider$1.class:na]

   3208         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider$1.perform(PluginAuthenticationProvider.java:94) ~[PluginAuthenticationProvider$1.class:na]

   3209         at com.atlassian.stash.internal.auth.DefaultCaptchaService.authenticateWithCaptcha(DefaultCaptchaService.java:71) ~[stash-service-impl-3.8.0.jar:na]

   3210         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider.attemptAuthentication(PluginAuthenticationProvider.java:114) [PluginAuthenticationProvider.class:na]

   3211         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider.authenticate(PluginAuthenticationProvider.java:61) [PluginAuthenticationProvider.class:na]

   3212         at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:97) [StashAuthenticationFilter.class:na]

   3213         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:111) [BeforeLoginPluginAuthenticationFilter.class:na]

   3214         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:77) [BeforeLoginPluginAuthenticationFilter.class:na]

   3215         at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100) [atlassian-trusted-apps-core-3.0.8.jar:na]

   3216         at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:79) [atlassian-oauth-service-provider-plugin-1.9.10_1421698664000.jar:na]

   3217         at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:32) [analytics-client-3.65_1427081260000.jar:na]

   3218         at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) [analytics-client-3.65_1427081260000.jar:na]

   3219         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:89) [BeforeLoginPluginAuthenticationFilter.class:na]

   3220         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75) [BeforeLoginPluginAuthenticationFilter.class:na]

   3221         at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:85) [stash-service-impl-3.8.0.jar:na]

   3222         at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38) [ConfigurableWebFilter.class:na]

   3223         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]

   3224         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

   3225         ... 182 frames trimmed

   3226 Caused by: com.atlassian.crowd.exception.runtime.OperationFailedException: null

   3227         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.convertOperationFailedException(CrowdServiceImpl.java:915) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3228         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:80) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3229         at com.atlassian.stash.internal.crowd.RiotPolice.authenticate(RiotPolice.java:98) ~[stash-service-impl-3.8.0.jar:na]

   3230         ... 21 common frames omitted

   3231 Caused by: com.atlassian.crowd.exception.ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. Go to JIRA home

   3232         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.throwError(RestExecutor.java:639) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3233         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:485) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3234         at com.atlassian.crowd.integration.rest.service.RestCrowdClient.authenticateUser(RestCrowdClient.java:139) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3235         at com.atlassian.crowd.directory.RemoteCrowdDirectory.authenticate(RemoteCrowdDirectory.java:194) ~[crowd-remote-2.8.2-m3-stash1.jar:na]

   3236         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:295) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3237         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:200) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3238         at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:283) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3239         at com.atlassian.stash.internal.crowd.CustomizedDirectoryManager.authenticateUser(CustomizedDirectoryManager.java:53) ~[stash-service-impl-3.8.0.jar:na]

   3240         at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:192) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3241         at com.atlassian.stash.internal.crowd.CustomizedApplicationService.authenticateUser(CustomizedApplicationService.java:46) ~[stash-service-impl-3.8.0.jar:na]

   3242         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:68) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3243         ... 22 common frames omitted

...skipping...

   3187         ... 21 common frames omitted

   3188 Caused by: com.atlassian.crowd.exception.ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. Go to JIRA home

   3189         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.throwError(RestExecutor.java:639) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3190         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:485) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3191         at com.atlassian.crowd.integration.rest.service.RestCrowdClient.authenticateUser(RestCrowdClient.java:139) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3192         at com.atlassian.crowd.directory.RemoteCrowdDirectory.authenticate(RemoteCrowdDirectory.java:194) ~[crowd-remote-2.8.2-m3-stash1.jar:na]

   3193         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:295) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3194         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:200) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3195         at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:283) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3196         at com.atlassian.stash.internal.crowd.CustomizedDirectoryManager.authenticateUser(CustomizedDirectoryManager.java:53) ~[stash-service-impl-3.8.0.jar:na]

   3197         at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:192) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3198         at com.atlassian.stash.internal.crowd.CustomizedApplicationService.authenticateUser(CustomizedApplicationService.java:46) ~[stash-service-impl-3.8.0.jar:na]

   3199         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:68) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3200         ... 22 common frames omitted

   3201 2015-11-16 17:55:52,138 ERROR [http-nio-7990-exec-7] @1C8BAPVx1075x32831x0 1h57xk4 67.244.86.216,127.0.0.1 "POST /j_stash_security_check HTTP/1.0" c.a.c.m.a.ApplicationServiceGeneric Directory 'JIRA User Directory' is not functional during authentication of 'quadr   3201 os'. Skipped.

   3202 2015-11-16 17:55:52,154 WARN  [http-nio-7990-exec-7] @1C8BAPVx1075x32831x0 1h57xk4 67.244.86.216,127.0.0.1 "POST /j_stash_security_check HTTP/1.0" c.a.s.i.s.s.PluginAuthenticationProvider Could not authenticate quadros; authentication by com.atlassian.stash.stash-   3202 authentication:crowdHttpAuthHandler failed

   3203 com.atlassian.stash.user.AuthenticationSystemException: The remote authentication server is not available. Please try again later.

   3204         at com.atlassian.stash.internal.crowd.RiotPolice.authenticate(RiotPolice.java:113) ~[stash-service-impl-3.8.0.jar:na]

   3205         at com.atlassian.stash.internal.user.DefaultUserService.authenticate(DefaultUserService.java:107) ~[stash-service-impl-3.8.0.jar:na]

   3206         at com.atlassian.stash.internal.auth.EmbeddedCrowdHttpAuthenticationHandler.authenticate(EmbeddedCrowdHttpAuthenticationHandler.java:40) ~[stash-service-impl-3.8.0.jar:na]

   3207         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider$1.perform(PluginAuthenticationProvider.java:97) ~[PluginAuthenticationProvider$1.class:na]

   3208         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider$1.perform(PluginAuthenticationProvider.java:94) ~[PluginAuthenticationProvider$1.class:na]

   3209         at com.atlassian.stash.internal.auth.DefaultCaptchaService.authenticateWithCaptcha(DefaultCaptchaService.java:71) ~[stash-service-impl-3.8.0.jar:na]

   3210         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider.attemptAuthentication(PluginAuthenticationProvider.java:114) [PluginAuthenticationProvider.class:na]

   3211         at com.atlassian.stash.internal.spring.security.PluginAuthenticationProvider.authenticate(PluginAuthenticationProvider.java:61) [PluginAuthenticationProvider.class:na]

   3212         at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:97) [StashAuthenticationFilter.class:na]

   3213         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:111) [BeforeLoginPluginAuthenticationFilter.class:na]

   3214         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:77) [BeforeLoginPluginAuthenticationFilter.class:na]

   3215         at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100) [atlassian-trusted-apps-core-3.0.8.jar:na]

   3216         at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:79) [atlassian-oauth-service-provider-plugin-1.9.10_1421698664000.jar:na]

   3217         at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:32) [analytics-client-3.65_1427081260000.jar:na]

   3218         at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32) [analytics-client-3.65_1427081260000.jar:na]

   3219         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:89) [BeforeLoginPluginAuthenticationFilter.class:na]

   3220         at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75) [BeforeLoginPluginAuthenticationFilter.class:na]

   3221         at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:85) [stash-service-impl-3.8.0.jar:na]

   3222         at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38) [ConfigurableWebFilter.class:na]

   3223         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]

   3224         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

   3225         ... 182 frames trimmed

   3226 Caused by: com.atlassian.crowd.exception.runtime.OperationFailedException: null

   3227         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.convertOperationFailedException(CrowdServiceImpl.java:915) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3228         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:80) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3229         at com.atlassian.stash.internal.crowd.RiotPolice.authenticate(RiotPolice.java:98) ~[stash-service-impl-3.8.0.jar:na]

   3230         ... 21 common frames omitted

   3231 Caused by: com.atlassian.crowd.exception.ApplicationPermissionException: Forbidden (403) Encountered a "403 - Forbidden" error while loading this page. Go to JIRA home

   3232         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.throwError(RestExecutor.java:639) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3233         at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:485) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3234         at com.atlassian.crowd.integration.rest.service.RestCrowdClient.authenticateUser(RestCrowdClient.java:139) ~[crowd-integration-client-rest-2.8.2-m3-stash1.jar:na]

   3235         at com.atlassian.crowd.directory.RemoteCrowdDirectory.authenticate(RemoteCrowdDirectory.java:194) ~[crowd-remote-2.8.2-m3-stash1.jar:na]

   3236         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticateAndUpdateInternalUser(DbCachingRemoteDirectory.java:295) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3237         at com.atlassian.crowd.directory.DbCachingRemoteDirectory.authenticate(DbCachingRemoteDirectory.java:200) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3238         at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:283) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3239         at com.atlassian.stash.internal.crowd.CustomizedDirectoryManager.authenticateUser(CustomizedDirectoryManager.java:53) ~[stash-service-impl-3.8.0.jar:na]

   3240         at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:192) ~[crowd-core-2.8.2-m3-stash1.jar:na]

   3241         at com.atlassian.stash.internal.crowd.CustomizedApplicationService.authenticateUser(CustomizedApplicationService.java:46) ~[stash-service-impl-3.8.0.jar:na]

   3242         at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:68) ~[embedded-crowd-core-2.8.2-m3-stash1.jar:na]

   3243         ... 22 common frames omitted

[Ariella Cheskis-Gold]

we have 0 users that can access stash right now

[Cat Peters]

Hi Nic, I'm also from the same company.

We don't know what steps to take to resolve this issue. Can you list some steps that we could take to resolve this?

We went through all of the groups that we changed and there doesn't appear to be any errors involved there. Do you think this could be related to groups or something else? We thought it may be caused by groups originally but we're not sure that's the case any longer.

[Nic Brough -Adaptavist-]

The log file says that Bitbucket cannot even *reach* the JIRA server to read the users. You need to fix that - it's either - A config error in Bitbucket - Your network not allowing the connection, - JIRA isn't running - JIRA is not allowing access to Bitbucket (I'm not sure you can do that with embedded Crowd)

[Queedref]

Jira is cloud based. Stash (Bitbucket) is hosted. Stash was working just fine previously, and works fine for anyone who HAS NOT signed out. Once you sign out, you cannot sign back in without receiving this error. What is embedded Crowd? As far as I know we don't have access to, nor are using crowd.

[Nic Brough -Adaptavist-]

The error message you've got is saying that Stash cannot connect to the user directory you have set up for it. At a glance, it looked like it was an embedded Crowd (that's the name for "An Atlassian application using a cut-down copy of Crowd internally to handle users) in JIRA, but re-reading the log, it might not be. Now you've said JIRA is Cloud based, that tells me a bit more. Your guess that changing JIRA groups is the problem is now a complete red-herring, because your Bitbucket Server won't be using Cloud JIRA for the user directories. You need to look at what user directory your Bitbucket Server is using and debug why it is unable to reach it over your network.

[Nirajan Thapa]

How do I look at which user directory my bitbucket server is using? Bear in mind I cannot log in to stash, but I have access to the server on which Stash is hosted. Once I find that directory, what should I be looking for?

[Nic Brough -Adaptavist-]

The user directory is handled inside the application, you need to log in as an admin to see it. Start at https://confluence.atlassian.com/bitbucketserver/lockout-recovery-process-776640158.html

[Nic Brough -Adaptavist-]

> The remote authentication server is not available

That answers the question - Bitbucket Server cannot reach the JIRA server to read the user details.

[Nic Brough -Adaptavist-]

If you think it's related to groups, then you need to compare the Stash (Well Bitbucket now) settings for groups with the changes you have made.