Bamboo hangs when running Mulesoft MUnit

[Phill Pafford]

Actually running Bamboo Data Center, the build agents are running in Docker on AWS ECS.

• I can run the build without the MUnit step and it runs fine
• If I add the MUnit step, I see MUnit start and just hangs after that

What I think is happening is MUnit is starting as an embedded server and the main process is waiting for MUnit to give back control/focus, as it looks like MUnit is a child process

example command without MUnit

/usr/local/bin/mvn clean deploy \
-DmuleDeploy \
-s maven/settings.xml 

example command with MUnit

/usr/local/bin/mvn clean deploy \
-DmuleDeploy \
-s maven/settings.xml \
-DskipMunitTests 

Logs

build 29-Jun-2023 15:48:46 [INFO] Running MULE_EE with version 4.4.0

build 29-Jun-2023 15:48:46 WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

build 29-Jun-2023 15:48:47 INFO 2023-06-29 11:48:47,344 [main] org.mule.munit.remote.RemoteRunner: Run Started

build 29-Jun-2023 15:48:47

build 29-Jun-2023 15:48:47

build 29-Jun-2023 15:48:47 INFO 2023-06-29 11:48:47,511 [main] org.mule.munit.remote.coverage.CoverageManager: Starting coverage server

build 29-Jun-2023 15:48:52 WARNING: An illegal reflective access operation has occurred

build 29-Jun-2023 15:48:52 WARNING: Illegal reflective access by org.mule.runtime.core.api.util.ClassUtils (file:/var/atlassian/application-data/bamboo-agent/.m2/repository/org/mule/runtime/mule-core/4.4.0/mule-core-4.4.0.jar) to constructor sun.net.www.protocol.https.Handler()

build 29-Jun-2023 15:48:52 WARNING: Please consider reporting this to the maintainers of org.mule.runtime.core.api.util.ClassUtils

build 29-Jun-2023 15:48:52 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

build 29-Jun-2023 15:48:52 WARNING: All illegal access operations will be denied in a future release

build 29-Jun-2023 15:48:52 INFO 2023-06-29 11:48:52,856 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:52 **********************************************************************

build 29-Jun-2023 15:48:52 * *

build 29-Jun-2023 15:48:52 * ((((((((((((((((((( *

build 29-Jun-2023 15:48:52 * ((((( ((((# *

build 29-Jun-2023 15:48:52 * &((( *((( *

build 29-Jun-2023 15:48:52 * &((( ((( *

build 29-Jun-2023 15:48:52 * ((( (((% (((# ((( *

build 29-Jun-2023 15:48:52 * (( ((((((( ((((((( ((( *

build 29-Jun-2023 15:48:52 * (( ((((((((( ((((((((( (( *

build 29-Jun-2023 15:48:52 * (( #((((((((((( #((((((((((( ((( *

build 29-Jun-2023 15:48:52 * #(( (((((((((((((( (((((((((((((( (( *

build 29-Jun-2023 15:48:52 * (( (((((( /((((((( ((((((( /(((((( (( *

build 29-Jun-2023 15:48:52 * (( (((((( (((((((&((((((( (((((( (( *

build 29-Jun-2023 15:48:52 * (( ((((( ((((((((((((* (((((( (( *

build 29-Jun-2023 15:48:52 * (( (((((( *((((((((( (((((( (( *

build 29-Jun-2023 15:48:52 * (( (((((( ((((((( (((((( ((( *

build 29-Jun-2023 15:48:52 * (( (((((( (((((( (( *

build 29-Jun-2023 15:48:52 * ((# ((((((# ((((((( (( *

build 29-Jun-2023 15:48:52 * ((# (((((((( (((((((( ((( *

build 29-Jun-2023 15:48:52 * ((( ((((((( ((((( ((. *

build 29-Jun-2023 15:48:52 * (( ((( ((( #(( *

build 29-Jun-2023 15:48:52 * /(( ((( *

build 29-Jun-2023 15:48:52 * ((( &((( *

build 29-Jun-2023 15:48:52 * *((((& &(((( *

build 29-Jun-2023 15:48:52 * ((((((((((((((/ *

build 29-Jun-2023 15:48:52 * *

build 29-Jun-2023 15:48:52 * ___ ___ _ ______ _ _ *

build 29-Jun-2023 15:48:52 * | \/ | | | | ___ \ | | (_) *

build 29-Jun-2023 15:48:52 * | . . |_ _| | ___ | |_/ / _ _ __ | |_ _ _ __ ___ ___ *

build 29-Jun-2023 15:48:52 * | |\/| | | | | |/ _ \ | / | | | '_ \| __| | '_ ` _ \ / _ \ *

build 29-Jun-2023 15:48:52 * | | | | |_| | | __/ | |\ \ |_| | | | | |_| | | | | | | __/ *

build 29-Jun-2023 15:48:52 * \_| |_/\__,_|_|\___| \_| \_\__,_|_| |_|\__|_|_| |_| |_|\___| *

build 29-Jun-2023 15:48:52 * *

build 29-Jun-2023 15:48:52 * *

build 29-Jun-2023 15:48:52 * Mule Runtime and Integration Platform *

build 29-Jun-2023 15:48:52 * Version: 4.4.0 Build: ee7cf916 *

build 29-Jun-2023 15:48:52 * MuleSoft, Inc. *

build 29-Jun-2023 15:48:52 * For more information go to *

build 29-Jun-2023 15:48:52 * https://www.mulesoft.com/platform/soa/mule-esb-enterprise *

build 29-Jun-2023 15:48:52 * *

build 29-Jun-2023 15:48:52 * Server started: 6/29/23, 11:48 AM *

build 29-Jun-2023 15:48:52 * JDK: 11.0.18 (mixed mode, sharing) *

build 29-Jun-2023 15:48:52 * JDK properties: *

build 29-Jun-2023 15:48:52 * - java.vendor = Eclipse Adoptium *

build 29-Jun-2023 15:48:52 * - java.vm.name = OpenJDK 64-Bit Server VM *

build 29-Jun-2023 15:48:52 * - java.home = /opt/java/openjdk *

build 29-Jun-2023 15:48:52 * OS: Linux (4.14.275-207.503.amzn2.x86_64, amd64) *

build 29-Jun-2023 15:48:52 * Host: ip-11-11-111-11.us-balrg-9.compute.internal (11-11-111-11) *

build 29-Jun-2023 15:48:52 * Mule services: *

build 29-Jun-2023 15:48:52 * - mule-service-weave-ee-2.4.0 *

build 29-Jun-2023 15:48:52 * - mule-service-oauth-ee-1.1.0 *

build 29-Jun-2023 15:48:52 * - mule-service-http-ee-1.7.0 *

build 29-Jun-2023 15:48:52 * - api-gateway-events-collector-service-1.1.0 *

build 29-Jun-2023 15:48:52 * - api-gateway-contract-service-1.2.0 *

build 29-Jun-2023 15:48:52 * - mule-service-soap-1.4.0 *

build 29-Jun-2023 15:48:52 * - mule-service-scheduler-1.4.0 *

build 29-Jun-2023 15:48:52 * Mule server plugins: *

build 29-Jun-2023 15:48:52 * - munit-coverage-plugin-2.3.14-mule-server-plugin.jar *

build 29-Jun-2023 15:48:52 * Mule system properties: *

build 29-Jun-2023 15:48:52 * - mule.testingMode = true *

build 29-Jun-2023 15:48:52 * - mule.nexus.username = My-Username.nexus *

build 29-Jun-2023 15:48:52 * - mule.env = dev *

build 29-Jun-2023 15:48:52 * - mule.home = *

build 29-Jun-2023 15:48:52 * /var/atlassian/application-data/bamboo-agent/xml-data/build-dir/NS *

build 29-Jun-2023 15:48:52 * FI-WSDD-JOB1/target/munitworkingdir-5449265435842150/container *

build 29-Jun-2023 15:48:52 * - mule.mode.embedded = true *

build 29-Jun-2023 15:48:52 * - mule.nexus.password = ***** *

build 29-Jun-2023 15:48:52 **********************************************************************

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,180 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * Cluster configuration *

build 29-Jun-2023 15:48:53 * *

build 29-Jun-2023 15:48:53 * Cluster mode disabled *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 WARN 2023-06-29 11:48:53,192 [Embedded] com.mulesoft.mule.runtime.plugin.discoverer.FileSystemPluginDiscoverer: Cannot unzip plugin file 'munit-coverage-plugin-2.3.14-mule-server-plugin.jar'

build 29-Jun-2023 15:48:53 WARN 2023-06-29 11:48:53,192 [Embedded] com.mulesoft.mule.runtime.plugin.discoverer.FileSystemPluginDiscoverer: Cannot delete plugin file 'munit-coverage-plugin-2.3.14-mule-server-plugin.jar'. Plugin will be overwritten on restart

build 29-Jun-2023 15:48:53 WARN 2023-06-29 11:48:53,192 [Embedded] com.mulesoft.mule.runtime.plugin.discoverer.FileSystemPluginDiscoverer: Cannot load descriptor for plugin 'munit-coverage-plugin-2.3.14-mule-server-plugin': Artifact descriptor does not exists: /var/atlassian/application-data/bamboo-agent/xml-data/build-dir/NSFI-WSDD-JOB1/target/munitworkingdir-5449265435842150/container/server-plugins/munit-coverage-plugin-2.3.14-mule-server-plugin/META-INF/mule-artifact/mule-artifact.json (org.mule.runtime.module.artifact.api.descriptor.ArtifactDescriptorCreateException)

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,217 [Embedded] org.mule.munit.plugins.coverage.CoverageServerPlugin: Initialising coverage plugin...

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,376 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 + New domain 'default' +

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,377 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 + Initializing domain 'default' +

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,850 [Embedded] org.mule.service.scheduler.internal.config.ContainerThreadPoolsConfig: No thread pools config file found. Using default values.

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,920 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * Started DefaultSchedulerService *

build 29-Jun-2023 15:48:53 * *

build 29-Jun-2023 15:48:53 * Resolved configuration values: *

build 29-Jun-2023 15:48:53 * *

build 29-Jun-2023 15:48:53 * Pooling strategy: UBER *

build 29-Jun-2023 15:48:53 * gracefulShutdownTimeout: 15000 ms *

build 29-Jun-2023 15:48:53 * uber.threadPool.maxSize: 164 *

build 29-Jun-2023 15:48:53 * uber.threadPool.threadKeepAlive: 30000 ms *

build 29-Jun-2023 15:48:53 * *

build 29-Jun-2023 15:48:53 * These can be modified by editing 'conf/scheduler-pools.conf' *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,924 [ArtifactDeployer.start.01] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * Domain: default *

build 29-Jun-2023 15:48:53 * OS encoding: UTF-8, Mule encoding: UTF-8 *

build 29-Jun-2023 15:48:53 * *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,925 [ArtifactDeployer.start.01] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * Started domain 'default' *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,929 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 + Mule is up and kicking (every 5000ms) +

build 29-Jun-2023 15:48:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:53 INFO 2023-06-29 11:48:53,931 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * - - + DOMAIN + - - * - - + STATUS + - - *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53 * default * DEPLOYED *

build 29-Jun-2023 15:48:53 **********************************************************************

build 29-Jun-2023 15:48:53

build 29-Jun-2023 15:48:54 INFO 2023-06-29 11:48:54,108 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:54 + New app 'my-sapi' +

build 29-Jun-2023 15:48:54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:54 INFO 2023-06-29 11:48:54,109 [Embedded] org.mule.runtime.core.internal.logging.LogUtil: 

build 29-Jun-2023 15:48:54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:54 + Initializing app 'my-sapi' +

build 29-Jun-2023 15:48:54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

build 29-Jun-2023 15:48:55 WARN 2023-06-29 11:48:55,728 [Embedded] org.mule.runtime.extension.internal.loader.enricher.ConfigRefDeclarationEnricher: Component 'console' in extension 'APIKit' already has a 'config-ref' parameter defined. Skipping ConfigRefDeclarationEnricher for it.

build 29-Jun-2023 15:48:55 WARN 2023-06-29 11:48:55,728 [Embedded] org.mule.runtime.extension.internal.loader.enricher.ConfigRefDeclarationEnricher: Component 'router' in extension 'APIKit' already has a 'config-ref' parameter defined. Skipping ConfigRefDeclarationEnricher for it.

it hangs for like 10/15 minutes then build fails

error	29-Jun-2023 12:09:53	Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.
simple	29-Jun-2023 12:09:53	Attempting to generate stack trace and terminate spawned sub-processes of process id: 9537
simple	29-Jun-2023 12:09:53	Found related process: pid: 19698 ppid: 9537 pgid: 9537 %cpu: 3.4 %mem: 1.4 cmd: /opt/java/openjdk/bin/java
simple	29-Jun-2023 12:09:53	Found related process: pid: 19751 ppid: 19698 pgid: 9537 %cpu: 0.0 %mem: 0.0 cmd: /bin/sh
simple	29-Jun-2023 12:09:53	Found related process: pid: 19752 ppid: 19751 pgid: 9537 %cpu: 1.9 %mem: 0.9 cmd: /opt/java/openjdk/bin/java
simple	29-Jun-2023 12:09:53	getStackTraceAndKillRelatedProcesses for 3 processes