SpringMVC dispatcher could not be started (Fatal error has ocurred)

Question

Hi, we have bitbucket 8.1.1 running on CentOS 7.9.

It did run ok until the last update of these 2 packages:git-1.8.3.1-23.el7_8 & perl-Git-1.8.3.1-23.el7_8 to git-1.8.3.1-23.el7_9 & perl-Git-1.8.3.1-23.el7_9.

Since bitbucket refuse to run with "Web server failed to start. Port 7990 was already in use."

The log file shows this:

2023-03-07 14:46:57,633 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.1.1 using Java 1.8.0_322 on engtools-rep with PID 4630 (/opt/atlassian/bitbucket/8.1.1/app/WEB-INF/classes started by root in /opt/atlassian/bitbucket/8.1.1/bin)
2023-03-07 14:46:57,639 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to 1 default profile: "default"
2023-03-07 14:46:58,626 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 8.1.1 (38bec1c built on Thu Jun 09 10:38:45 BST 2022)
2023-03-07 14:46:58,626 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Temurin OpenJDK 64-Bit Server VM 1.8.0_322-b06
2023-03-07 14:47:00,043 WARN [main] o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.PortInUseException: Port 7990 is already in use
2023-03-07 14:47:00,043 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Bitbucket 8.1.1 has shut down
2023-03-07 14:47:00,530 WARN [main] o.a.c.loader.WebappClassLoaderBase The web application [ROOT] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.management.NotificationEmitterSupport.removeNotificationListener(NotificationEmitterSupport.java:109)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaserListener.unregister(BlockSource.java:413)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:335)
2023-03-07 14:47:00,603 ERROR [main] o.s.b.d.LoggingFailureAnalysisReporter

***************************
APPLICATION FAILED TO START
***************************

Description:

Web server failed to start. Port 7990 was already in use.

Action:

Identify and stop the process that's listening on port 7990 or configure this application to listen on another port.

2023-03-07 14:47:18,525 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,527 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,527 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,528 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,528 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,528 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,529 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,533 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,533 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,534 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,534 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,534 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,535 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,535 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:18,535 WARN [http-nio-7990-exec-9] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,968 WARN [http-nio-7990-exec-10] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,969 WARN [http-nio-7990-exec-10] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,983 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [hazelcastFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [pluginProvidedAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [goGetFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,984 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,986 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,986 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [requestAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [entityMovedFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [trustedApplicationsAuthenticationFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [springSecurityFilterChain]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [authenticationAttributeFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [rateLimitFilter]; no Spring WebApplicationContext is available
2023-03-07 14:47:23,987 WARN [http-nio-7990-exec-1] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing [standardResponseHeadersFilter]; no Spring WebApplicationContext is available
2023-03-07 15:17:39,625 WARN [Catalina-utility-1] com.hazelcast.web.SessionListener The com.hazelcast.web.WebFilter could not be found. com.hazelcast.web.SessionListener should be paired with a com.hazelcast.web.WebFilter.

I tried downgrading the packages with the same resolution. It seems to be a configuration issue but I can't see how the update could have affected it.

Any idea  how to resolve this problem?

I'm grateful for any suggestion.

Christian Glockner · Answer

Hi Joe,

5701 belongs to the Hazelcast part of Bitbucket Server, so no wonder it shuts down if you close that port.

From the earlier messages it seems like Bitbucket Server actually starts fine, but it's the Opensearch part that fails to start because you're starting it as root (which is not a good idea for Bitbucket Server too, by the way). Try starting it as a dedicated user (the default username is atlbitbucket) to see if that makes a difference.

Cheers,

Christian

Premier Support Engineer

Atlassian

Joe Powell · Answer

Hi Christian,

just add to another attempt to sort out this problem, I stop all java processes ( Bitbucket,Jira,Confluence ) and start bitbucker again. Now I check the java process to make sure it was running and got this result: "  netstat -tlpn | grep java
tcp6 0 0 :::7990 :::* LISTEN 2846/java
tcp6 0 0 :::5701 :::* LISTEN 2846/java "

If I stop 5701, automatically close 7990.

Could that be part of the problem?

Thanks,

Joe Powell · Answer

Hi Christian,

Thanks for you response.

Yes that was on of the first thing I tried without success.

I tried it again today with following oputput:

netstat -ltnp | grep java
tcp6 0 0 :::8090 :::* LISTEN 2827/java 
tcp6 0 0 :::8091 :::* LISTEN 3277/java 
tcp6 0 0 127.0.0.1:8000 :::* LISTEN 2827/java 
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 1490/java 
tcp6 0 0 :::8080 :::* LISTEN 1490/java 
[root@engtools-rep ~]# cd /opt/atlassian/bitbucket/8.1.1/bin/
[root@engtools-rep bin]# ./start-bitbucket.sh 
Starting Atlassian Bitbucket as the current user

Starting the bundled search server
Hint: Run start-bitbucket.sh --no-search to skip starting the bundled search server
future versions of OpenSearch will require Java 11; your Java version from [/opt/atlassian/bitbucket/8.1.1/jre] does not meet this requirement
future versions of OpenSearch will require Java 11; your Java version from [/opt/atlassian/bitbucket/8.1.1/jre] does not meet this requirement
Bundled search server started successfully

Bitbucket is being run with a umask that contains potentially unsafe settings.
The following issues were found with the mask "u=rwx,g=rx,o=rx" (0022):
- Access is allowed to 'others'. It is recommended that 'others' be denied
all access for security reasons.
The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be
configured in _start-webapp.sh

The current open files limit is set to less than 6192
Attempting to increase limit...
Limit increased to 6192 open files

Starting Bitbucket webapp at http://localhost:7990
The Bitbucket webapp has been started.

If you cannot access Bitbucket within 3 minutes, or encounter other issues, check the troubleshooting guide at:
https://go.atlassian.com/bbs-troubleshooting-installations
[root@engtools-rep bin]# uncaught exception in thread [main]
java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /var/atlassian/application-data/bitbucket/log/search/bitbucket_search.log

I found in your webpage similar problem in case 5 "https://confluence.atlassian.com/bitbucketserverkb/springmvc-dispatcher-could-not-be-started-779171477.html"

With following resolution:

* Resolution for cause #5:The error message states unsatisfied dependancy exception for GIT version on the Bitbucket server.  On Some of the latest linux distributions GIT version is 2.38.x or later, which is still not supported on Bitbucket 8.6.x or below and will likely Bitbucket fails to start. Please refer Supported platform page for your Bitbucket version and compile a supported version of GIT to resolve the issue. 
* When I revert the changes via snapshot, it works fine. If I only undo the package updates it stay broken. 
* Another thing I can't understand and it might not be relevant, is the message about OpenSearch will require Java 11; your Java version from [/opt/atlassian/bitbucket/8.1.1/jre] does not meet this requirement future versions of OpenSearch will require Java 11.

Thanks for your support.

Joe.