failure when install bitbucket pipeline runners

[William_ Wang Lei]

[2021-06-24 02:42:42,561] Runner starting version: 1.146
[2021-06-24 02:42:44,321] /tmp/libnetty_transport_native_epoll_x86_643706708993218820314.so exists but cannot be executed even when execute permissions set; check volume for "noexec" flag; use -Dio.netty.native.workdir=[path] to set native working directory separately.
[2021-06-24 02:42:44,324] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dockerServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/DockerServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'containerServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/ContainerServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dockerClient' defined in com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library
Exception in thread "main" org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dockerServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/DockerServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'containerServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/ContainerServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dockerClient' defined in com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
at com.atlassian.pipelines.runner.core.ApplicationImpl.main(ApplicationImpl.java:44)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'containerServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/ContainerServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dockerClient' defined in com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
... 13 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dockerClient' defined in com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:635)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
... 27 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650)
... 41 more
Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library
at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:80)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:51)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:150)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:112)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:99)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:76)
at com.github.dockerjava.netty.NettyDockerCmdExecFactory$UnixDomainSocketInitializer.epollGroup(NettyDockerCmdExecFactory.java:152)
at com.github.dockerjava.netty.NettyDockerCmdExecFactory$UnixDomainSocketInitializer.init(NettyDockerCmdExecFactory.java:144)
at com.github.dockerjava.netty.NettyDockerCmdExecFactory.init(NettyDockerCmdExecFactory.java:115)
at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:242)
at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:93)
at com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl.dockerClient(SpringConfigurationImpl.java:117)
at com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl$$EnhancerBySpringCGLIB$$fdf8b339.CGLIB$dockerClient$8(<generated>)
at com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl$$EnhancerBySpringCGLIB$$fdf8b339$$FastClassBySpringCGLIB$$ff3a09e5.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl$$EnhancerBySpringCGLIB$$fdf8b339.dockerClient(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 42 more
Caused by: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643706708993218820314.so: /tmp/libnetty_transport_native_epoll_x86_643706708993218820314.so: failed to map segment from shared object
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.load0(Unknown Source)
at java.base/java.lang.System.load(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:351)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:250)
at io.netty.channel.epoll.Native.<clinit>(Native.java:69)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
... 68 more
Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_transport_native_epoll_x86_643706708993218820314.so: /tmp/libnetty_transport_native_epoll_x86_643706708993218820314.so: failed to map segment from shared object
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.load0(Unknown Source)
at java.base/java.lang.System.load(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:371)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:363)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:341)
... 72 more
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
at java.base/java.lang.System.loadLibrary(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:351)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 71 more
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
at java.base/java.lang.System.loadLibrary(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:371)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:363)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:341)
... 72 more
Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:226)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:253)
... 70 more
Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:175)
... 71 more
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
at java.base/java.lang.System.loadLibrary(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:351)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 71 more
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
at java.base/java.lang.System.loadLibrary(Unknown Source)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:371)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:363)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:341)
... 72 more

[Alban Peignier]

This error can be experienced when the host os use the flag noexec on the /tmp directory.

$ mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,noexec,nosuid,noatime)

The problem can be prevented by using another work directory (as described in the Runner documentation) where the noexec flag is not used.

We're using a dedicated tmpfs directory on our VMs:

$ sudo mkdir /var/lib/bitbucket-pipelines-runner 
$ sudo mount -t tmpfs tmpfs /var/lib/bitbucket-pipelines-runner

$ docker container run -it -v /var/lib/bitbucket-pipelines-runner:/var/lib/bitbucket-pipelines-runner \ 
 -v /var/run/docker.sock:/var/run/docker.sock [...] \ 
 -e WORKING_DIRECTORY=/var/lib/bitbucket-pipelines-runner --name runner \ 
 docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner