Bitbucket 7.21.3 Windows installer problem

[David Tupper]

The problem in our production environment is though the Bitbucket 7.21.3 Windows installer did create the Bitbucket Server Windows service, the installer failed to create the Windows service for search (OpenSearch).  I am perplexed because I ran the same process in test without issue. I have attempted to install the Windows service for search manually.  The Windows service for search is created with a warning. Still the Windows service for search will not start.  I need assistance resolving this problem.

The envirionment for test mirrors the production environment.
--The Bitbucket database is using SQL Server 2016 and is hosted on a Windows Server 2016 that is separate from the Atlassian application server.
--The Atlassian applications (Jira, Confluence, Bitbucket, and Crowd) are running as Windows services on a Windows Server 2012.
--I upgraded Bitbucket on the test server from 6.10 to 7.21.3 using the Windows installer. The upgrade was a success and the installer replaced the ElasticSearch Windows service with an OpenSearch Windows service and successfully started the OpenSearch and Bitbucket server Windows services.
--I upgraded Bitbucket on the production server from 6.10 to 7.21.3 using the Windows installer. The upgrade was partially successful. The Bitbucket server Windows service is installed and operational. The ElasticSearch Windows service was not replaced with an OpenSearch Windows service. My attempt to manually install the OpenSearch Windows service gives a warning and the installed service will not start. Why did the process that worked in test fail in production? How can I resolve the issue?

[Prashant Mulya · Accepted solution]

Hi David,

So far I understand the current status as below

- Bitbucket Service is installed and running fine 

- OpenSearch Service is installed however it is unable to start due to the error seen in the search logs

The error indicates that OpenSearch is unable to find the certificate under the config folder.

From the Error, it looks like Open search config is folder set to E:\bitbucket_install\opensearch\config\ folder in the Java Arguments and Certificates doesn't exist under the folder

You should be able to find the following 3 certificates on the server

-rw------- 1 atlbitbucket atlbitbucket 1009 Mar 3 09:12 root-ca.pem
-rw------- 1 atlbitbucket atlbitbucket 981 Mar 3 09:12 bitbucket.pem
-rw------- 1 atlbitbucket atlbitbucket 1704 Mar 3 09:12 bitbucket-key.pem

Generally, the search config folder is set to Bitbucket-home\shared\search\config. Please check to see if you see the certs here

Once you have found the above 3 certificates in the server you can either

- copy them over to the E:\bitbucket_install\opensearch\config\
- Or you can change the Opensearch_config folder to the folder where the certificate exists by updating the Java Arguments for the OpenSearch Service. Here is a sample KB on how to check Java Arguments for Windows Service.

And Finally, you may have to follow the process listed in the Article Test button in the Search server for Bitbucket server results in the access denied in case you encounter a similar error after the OpenSearch service starts fine.

I hope this helps

Regards,

Prashant

[David Tupper]

Thank you Prashant!

Your suggestion (to change the Opensearch_config folder to the folder where the certificate exists by updating the Java Arguments for the OpenSearch Service) allowed me to resolve the problem.

1) I used the Windows command prompt to edit the OpenSearch service.

E:\bitbucket_install\opensearch\bin>opensearch-service-mgr.exe //ES/OpenSearch

2) On the Java tab, I changed the Java Options section to use the path where the certs were found in my environment.

-Dopensearch.path.conf=E:\bitbucket_home\shared\search\config

[drewdemarco]

Hi @Prashant Mulya 

On the Running Bitbucket Server as a Windows service page it describes step 8 as

If you are installing a new instance of Bitbucket Server, are upgrading from Bitbucket Server 4.5 to Bitbucket 4.6 to 7.20, or are upgrading from Bitbucket 7.20 to Bitbucket 7.21, copy the template OpenSearch configuration files over to your Bitbucket home directory by running the following command:

robocopy /S config-template %BITBUCKET_HOME%\shared\search
But on a clean install of 7.21.4 there is not config-template folder that I can see within <Bitbucket Server installation directory>\opensearch 

It seems like your instructions should be what's listed as what to do. As the pem and config files seem to be created within the %BITBUCKET_HOME% by default with the install

- copy them over to the E:\bitbucket_install\opensearch\config\
- Or you can change the Opensearch_config folder to the folder where the certificate exists by updating the Java Arguments for the OpenSearch Service. here is a sample KB on how to check Java Arguments for Windows Service.

[Charlie Misonne]

Hi David,

Can you tell us more about the warning you are getting when the service is created?
And do you get anything in the Bitbucket logs when you attempt to start the service? You can also check the Windows event log.

Remember you can also create an Atlassian support request if this issue is crtitical to you: https://support.atlassian.com/contact. Of course you need a valid license of Bitbucket.

[David Tupper]

Hello Charlie,

Thank you for the reply.  Any help you may provide is appreciated given that I only have a starter license for Bitbucket.

I tried the new Bitbucket 7.21.4 (64 bit Windows) installer in both test and production to see what would happen.  What perplexes me is the difference in behavior between test and production.  Running the installer in the test enivronment automatically updated the Windows services:

1) Removed the Elasticsearch Windows service

2) Installed the OpenSearch Windows service

3) Updated the Bitbucket Windows service description to reflect the latest version information (e.g. first Atlassian Bitbucket [7.21.3] and then Atlassian Bitbucket [7.21.4].

I have no problems with the Windows services in the test envirionment.

Running the installer in the production environment does not automatically update the corresponding Windows services.  Is there potentially a configuration item I am missing that is causing the difference in installer behavior in production?  I have to use the command prompt to install the Windows services in production (https://confluence.atlassian.com/bitbucketserver0719/running-bitbucket-server-as-a-windows-service-1108482512.html).

Below is the output from installing the OpenSearch Windows service.

E:\bitbucket_install\opensearch\bin>service.bat install
"warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release"
future versions of OpenSearch will require Java 11; your Java version from [C:\Java\jdk1.8.0_91\jre] does not meet this requirement
Installing service : "BITBUCKET_SEARCH"
Using JAVA_HOME (64-bit): "C:\Java\jdk1.8.0_91"
Error: Could not find or load main class com.atlassian.bitbucket.searchconfig.SearchConfigCli
Warning: with JDK 8 on Windows, OpenSearch may be unable to derive correct ergonomic settings due to a JDK issue (JDK-8074459). Please use a newer version of Java.
Warning: MaxDirectMemorySize may have been miscalculated due to JDK-8074459. Please use a newer version of Java or set MaxDirectMemorySize explicitly.
-Dopensearch.networkaddress.cache.ttl=60;-Dopensearch.networkaddress.cache.negative.ttl=10;-XX:+AlwaysPreTouch;-Xss1m;-Djava.awt.headless=true;-D file.encoding=UTF-8;-Djna.nosys=true;-XX:-OmitStackTraceInFastThrow;-Dio.netty.noUnsafe=true;-Dio.netty.noKeySetOptimization=true;-Dio.netty.recycler.maxCapacityPerThread=0;-Dio.netty.allocator.numDirectArenas=0;-Dlog4j.shutdownHookEnabled=false;-Dlog4j2.disable.jmx=true;-Djava.locale.providers=SPI,JRE;-Xms1g;-Xmx1g;-XX:+UseConcMarkSweepGC;-XX:CMSInitiatingOccupancyFraction=75;-XX:+UseCMSInitiatingOccupancyOnly; -Djava.io.tmpdir=C:\Users\dt13095\AppData\Local\Temp\opensearch;-XX:+HeapDumpOnOutOfMemoryError;-XX:HeapDumpPath=data;-XX:ErrorFile=logs/hs_err_pid%p.log;-XX:+PrintGCDetails;-XX:+PrintGCDateStamps;-XX:+PrintTenuringDistribution;-XX:+PrintGCApplicationStoppedTime;-Xloggc:logs/gc.log;-XX:+UseGCLogFileRotation;-XX:NumberOfGCLogFiles=32;-XX:GCLogFileSize=64m;-XX:MaxDirectMemorySize=536870912
The service 'BITBUCKET_SEARCH' has been installed.

[David Tupper]

Hi Charlie,

In checking the E:\bitbucket_home\log\search\bitbucket_search.log in test and production I noticed a configuration difference as well as an error in the production log.

Test Config and no errors in the log:

OpenSearch Config path is E:\bitbucket_home\shared\search\config

Config directory is E:\bitbucket_home\shared\search\config/

Production Config and error in the log:

OpenSearch Config path is E:\bitbucket_install\opensearch\config

Config directory is E:\bitbucket_install\opensearch\config/

Error:  org.opensearch.OpenSearchException: Unable to read E:\bitbucket_install\opensearch\config\bitbucket.pem (E:\bitbucket_install\opensearch\config\bitbucket.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath

Do you have a recommendation on how to resolve this issue?

Thank you

[drewdemarco]

@David Tupper  were you ever able to find a resolution to this issue?

Thanks.

[David Tupper]

I haven't been able to resolve this issue yet.

[Alex Skobelev]

In a recent upgrade to LTS 7.21.7 we ran into this issue, where log errors showing unable to find the bitbucket.pem/bitbucket-key.pem/root-ca.pem in the opensearch directory.

We were able to resolve this by pulling these certificates from the previous installation directory (found them by searching the root install directory) and copied them into the new expected directory as mentioned in the logs (c:\bitbucket_install\opensearch\config)

Once the certificates were in place, the search service started up and worked without issue.

Hope this helps!