Repositories Not Showing After Upgrade

[Dean Martens]

I upgraded from Stash 3.3 to BitBucket Server 5.10.  When I log into BitBucket, my repositories are not showing; it just shows a blank page.

I've upgraded in an environment without internet access.  I've focused on the bitbucket properties for quite a while now and haven't made any progress.  I've installed and re-installed many, many times trying different things.  I'm completely out of ideas.  Below is the log entries and previous config and current config with regard to the connector.  Sensitive names replaced by {dbserver} and ********:

#>*******************************************************
#> Migrated to database at jdbc:sqlserver://{dbserver}:1433;databaseName=STASH;
#> Updated by Anonymous on 2013-04-01T13:56:51.061-07:00

#>*******************************************************
jdbc.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbc.url=jdbc:sqlserver://{dbserver}:1433;databaseName=STASH;
jdbc.user=Stash_User
jdbc.password=*******
server.port=443
server.secure=true
server.scheme=https
server.ssl.enabled=true
server.ssl.client-auth=want
server.ssl.protocol=TLSv1.2
server.ssl.key-store=c:\\java\\bin\\tomcat.keystore
server.ssl.key-store-password=******
server.ssl.key-password=*******

Log:

2018-04-27 14:50:40,505 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on AWSOM-O with PID 5096 (C:\Atlassian\Bitbucket\5.10.0\app\WEB-INF\classes started by system1 in C:\Atlassian\Bitbucket\5.10.0\bin)
2018-04-27 14:50:40,505 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2018-04-27 14:50:43,110 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 5.10.0 (76ff11a built on Sun Apr 22 19:39:21 PDT 2018)
2018-04-27 14:50:43,110 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_171-b11
2018-04-27 14:50:48,960 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 12.511 seconds (JVM running for 23.269)
2018-04-27 14:50:57,774 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory C:\Stash_Data
2018-04-27 14:51:01,456 INFO [spring-startup] c.a.s.internal.ApplicationSettings Updated home version from 3.3.0 to 5.10.0
2018-04-27 14:51:12,672 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory C:\Stash_Data\shared
2018-04-27 14:51:17,758 WARN [spring-startup] com.hazelcast.instance.Node [10.0.15.9]:5701 [system1] [3.7.4-atlassian-43] No join method is enabled! Starting standalone.
2018-04-27 14:52:13,139 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:49328
2018-04-27 14:52:18,099 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:49329
2018-04-27 14:52:21,547 INFO [git-upgrade] c.a.s.i.s.g.u.s.SalGitUpgradeManager All asynchronous upgrade tasks have completed
2018-04-27 14:53:02,049 WARN [spring-startup] c.a.s.i.plugin.PluginUpgradeHelper Disabled com.lb.software.stash.jira.connector.lb-software-stash-jira-connector
2018-04-27 14:53:13,312 WARN [ThreadPoolAsyncTaskExecutor::Thread 9] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2018-04-27 14:53:24,312 WARN [ThreadPoolAsyncTaskExecutor::Thread 1] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2018-04-27 14:53:26,683 WARN [ThreadPoolAsyncTaskExecutor::Thread 8] c.a.p.s.s.util.ProductFilterUtil Couldn't detect product, will use ProductFilter.ALL
2018-04-27 14:53:50,476 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2018-04-27 14:53:52,441 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2018-04-27 14:53:52,472 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2018-04-27 14:53:52,488 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2018-04-27 14:53:52,488 INFO [spring-startup] c.a.s.i.plugin.PluginUpgradeHelper Disabled 1 plugins during Bitbucket upgrade
2018-04-27 14:53:54,079 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2018-04-27 14:53:54,079 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2018-04-27 14:53:54,095 INFO [spring-startup] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for ElasticsearchSynchronizeJob
2018-04-27 14:53:54,095 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2018-04-27 14:53:54,235 INFO [spring-startup] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2018-04-27 14:53:54,344 INFO [spring-startup] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2018-04-27 14:53:54,610 INFO [spring-startup] c.a.s.internal.scm.PluginScmService Bitbucket is using C:\Program Files (x86)\Git\cmd\git.exe version 2.16.2
2018-04-27 14:53:54,688 INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Updating Hazelcast instance capabilities
2018-04-27 14:54:04,110 INFO [Caesium-1-4] c.a.b.i.s.i.jobs.StartupChecksJob Attempting provisioning in startup job
2018-04-27 14:54:04,110 INFO [Caesium-1-4] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2018-04-27 14:54:04,344 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Running up to 3 search health checks
2018-04-27 14:54:04,469 ERROR [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in Elasticsearch
2018-04-27 14:54:04,469 WARN [Caesium-1-4] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: 4b552e44-5c88-4655-8a1f-a46a23821fb3
2018-04-27 14:54:04,484 INFO [I/O dispatcher 5] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Testing connection with Elasticsearch failed: HTTP response status code is 401
2018-04-27 14:54:04,562 INFO [I/O dispatcher 9] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2018-04-27 14:54:04,562 INFO [Caesium-1-1] c.a.b.i.s.c.s.p.AutomaticAuthenticationProvisioner Elasticsearch is using default credentials. Will attempt to provision authentication
2018-04-27 14:54:04,687 INFO [I/O dispatcher 17] c.a.b.s.s.t.DefaultElasticsearchConnectionTester Connected to Elasticsearch version 5.5.3
2018-04-27 14:54:04,750 INFO [Caesium-1-1] c.a.b.i.s.c.s.p.AutomaticAuthenticationProvisioner Authentication was failing, but Elasticsearch was set to default password. Reattempting original provisioning should now occur through StartupChecks
2018-04-27 14:54:04,750 INFO [Caesium-1-1] c.a.b.i.s.i.j.c.AuthenticationCheck Authentication was failing, but Elasticsearch was configured to use the default installer password.Reprovisioning has taken place and Bitbucket Search will now resynchronize.
2018-04-27 14:54:04,812 INFO [Caesium-1-1] c.a.b.i.s.c.c.DefaultClusterJobManager Attempting to schedule an ad-hoc for ElasticsearchSynchronizeJob
2018-04-27 14:54:04,812 INFO [Caesium-1-1] c.a.b.i.s.c.c.DefaultClusterJobManager There are currently 2 to select from
2018-04-27 14:54:04,874 INFO [Caesium-1-4] c.a.b.i.s.i.IndexingSynchronizationService Beginning indexing of all projects and repositories for search.
2018-04-27 14:54:04,874 INFO [Caesium-1-1] c.a.b.i.s.c.j.c.HealthCheckRunner Health checks will run again after PT5M
2018-04-27 14:54:05,420 WARN [UpmScheduler:thread-2] com.atlassian.upm.pac.PacClientImpl Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: java.net.UnknownHostException: marketplace.atlassian.com
2018-04-27 14:54:05,467 WARN [Caesium-1-2] c.a.a.c.DefaultApplinkStatusService Unrecognized error while attempting to retrieve status of Application Link '61b6191d-d412-3043-a96c-75b7bceaed1f'
2018-04-27 14:54:16,670 WARN [Caesium-1-3] c.a.a.c.h.r.RemoteHashingInstructionsReader Unable to read remote instructions with key 'uid.onewayhash'.
2018-04-27 14:54:16,670 WARN [Caesium-1-3] c.a.a.c.h.BcryptAnalyticsEmailHasher No instructions for hashing could be found.

[sandeep_sankalapur]

We had similar issue post upgrade, Disabling all plugins started to show up repositories.
Later on we found the problematic plugin & removed it.

[Mai Saleh]

We have same issue after upgrade, please advise how did you fix it.

[Felipe Kraemer]

Hi @Dean Martens,

There are no errors in the log excerpt you have provided, so it's hard to tell what is going on.

You have mentioned the problem happens after logging into the instance, so as far as I understand the instance itself is reachable in the web browser at the usual URL and you can login as expected, but after that you don't see any repositories in your dashboard.

This suggests that the connector is configured correctly to use the SSL certificate in bitbucket.properties, but if you used to run Stash 3.3 behind a proxy, I noticed that the proxy settings are missing in your bitbucket.properties.

When you installed Bitbucket Server, have you configured it to start as Windows Service? If that is correct, can you please check if the Windows service is using a dedicated user account?

If you start Bitbucket Server manually, have you started it with the same Windows user that used to start Stash 3.3?

It might be better to create a support request so that the instance settings and configuration files can be investigated further.

Regards,

Felipe