Bamboo - Bitbucket issue

[Víctor Almaraz]

Hi everybody!

I am just writing in case there is somebody in the same situation.

We have a Bamboo instance (6.2.2) in one of our servers (Debian) and it is connected to a cloud Bitbucket account.

Since Jan 9, when Bitbucket starts to fail responding requests (https://status.bitbucket.org/), our Bamboo instance failed and we are unable to start up again.

After restarting the Bamboo service, it seems to work, but I think that it fails when it tries to contact Bitbucket repositories.

Every time we try to start the service, after a couple of minutes, we can read this message:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fd4984ca547, pid=26187, tid=0x00007fd3f463c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 20205 C2 org.bouncycastle.crypto.engines.AESFastEngine.encryptBlock([[I)V (1795 bytes) @ 0x00007fd4984ca547 [0x00007fd4984c9fa0+0x5a7]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /usr/local/bamboo/hs_err_pid26187.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

As it is related with cryptography, and it starts just when Bitbucket went down, we think it could be linked to some change or something malfunctioning on Bitbucket.

Is this a known bug? Something we have to update? Or simply wait until Bitbucket restore the storage issue?

Thanks a lot!

PD: I paste here the stack trace from the error report:

Stack: [0x00007fd3f453c000,0x00007fd3f463d000], sp=0x00007fd3f46375a0, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 20205 C2 org.bouncycastle.crypto.engines.AESFastEngine.encryptBlock([[I)V (1795 bytes) @ 0x00007fd4984ca547 [0x00007fd4984c9fa0+0x5a7]
J 20245 C2 org.bouncycastle.crypto.modes.SICBlockCipher.calculateByte(B)B (95 bytes) @ 0x00007fd4984ce750 [0x00007fd4984ce300+0x450]
J 21289 C2 org.bouncycastle.crypto.BufferedBlockCipher.processBytes([BII[BI)I (242 bytes) @ 0x00007fd49885c4d0 [0x00007fd49885c0a0+0x430]
J 21656 C2 org.apache.sshd.common.session.helpers.AbstractSession.encode(Lorg/apache/sshd/common/util/buffer/Buffer;)V (527 bytes) @ 0x00007fd496dc75c8 [0x00007fd496dc6a20+0xba8]
J 21652 C2 org.apache.sshd.common.channel.AbstractChannel.writePacket(Lorg/apache/sshd/common/util/buffer/Buffer;)Lorg/apache/sshd/common/io/IoWriteFuture; (54 bytes) @ 0x00007fd49872a3bc [0x00007fd49872a180+0x23c]
J 21657 C2 org.apache.sshd.common.channel.ChannelOutputStream.flush()V (536 bytes) @ 0x00007fd496dc010c [0x00007fd496dbf640+0xacc]
J 21634 C2 org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(Lorg/apache/sshd/common/util/buffer/Buffer;)V (326 bytes) @ 0x00007fd4981dbd18 [0x00007fd4981d8b60+0x31b8]
J 21608 C2 org.apache.sshd.common.session.helpers.AbstractSession.decode()V (742 bytes) @ 0x00007fd498569de8 [0x00007fd498569b60+0x288]
J 21524 C2 org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run()Ljava/lang/Object; (17 bytes) @ 0x00007fd49857e8ac [0x00007fd49857db60+0xd4c]
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72d16c] JVM_DoPrivileged+0x27c
J 685 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fd49530883f [0x00007fd495308780+0xbf]
J 21640 C2 sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(ZLjava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;JLjava/util/concurrent/TimeUnit;Ljava/lang/Object;Ljava/nio/channels/CompletionHandler;)Ljava/util/concurrent/Future; (429 bytes) @ 0x00007fd4983898a0 [0x00007fd4983892c0+0x5e0]
J 21524 C2 org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run()Ljava/lang/Object; (17 bytes) @ 0x00007fd49857ea68 [0x00007fd49857db60+0xf08]
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72d16c] JVM_DoPrivileged+0x27c
J 685 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fd49530883f [0x00007fd495308780+0xbf]
J 21640 C2 sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(ZLjava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;JLjava/util/concurrent/TimeUnit;Ljava/lang/Object;Ljava/nio/channels/CompletionHandler;)Ljava/util/concurrent/Future; (429 bytes) @ 0x00007fd4983898a0 [0x00007fd4983892c0+0x5e0]
J 21524 C2 org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run()Ljava/lang/Object; (17 bytes) @ 0x00007fd49857ea68 [0x00007fd49857db60+0xf08]
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72d16c] JVM_DoPrivileged+0x27c
J 685 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fd49530883f [0x00007fd495308780+0xbf]
J 21640 C2 sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(ZLjava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;JLjava/util/concurrent/TimeUnit;Ljava/lang/Object;Ljava/nio/channels/CompletionHandler;)Ljava/util/concurrent/Future; (429 bytes) @ 0x00007fd4983898a0 [0x00007fd4983892c0+0x5e0]
J 21524 C2 org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run()Ljava/lang/Object; (17 bytes) @ 0x00007fd49857ea68 [0x00007fd49857db60+0xf08]
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72d16c] JVM_DoPrivileged+0x27c
J 685 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fd49530883f [0x00007fd495308780+0xbf]
J 21640 C2 sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(ZLjava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;JLjava/util/concurrent/TimeUnit;Ljava/lang/Object;Ljava/nio/channels/CompletionHandler;)Ljava/util/concurrent/Future; (429 bytes) @ 0x00007fd4983898a0 [0x00007fd4983892c0+0x5e0]
J 21524 C2 org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run()Ljava/lang/Object; (17 bytes) @ 0x00007fd49857ea68 [0x00007fd49857db60+0xf08]
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72d16c] JVM_DoPrivileged+0x27c
J 685 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fd49530883f [0x00007fd495308780+0xbf]
J 21632 C2 sun.nio.ch.AsynchronousChannelGroupImpl$1.run()V (17 bytes) @ 0x00007fd498427328 [0x00007fd498426fe0+0x348]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x693e96] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x6943a1] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x694847] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x72fbd0] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa79753] JavaThread::thread_main_inner()+0x103
V [libjvm.so+0xa7989c] JavaThread::run()+0x11c
V [libjvm.so+0x92a338] java_start(Thread*)+0x108
C [libpthread.so.0+0x8064] start_thread+0xc4

Some events info:

Internal exceptions (10 events):
Event: 144.429 Thread 0x00007fd42971a800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f1f939f0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 146.108 Thread 0x00007fd42946d800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f1700608) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 146.431 Thread 0x00007fd42971a800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f1f93fe8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 148.109 Thread 0x00007fd42946d800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f17009a0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 148.434 Thread 0x00007fd42971a800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f8486798) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 150.112 Thread 0x00007fd42946d800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x0000000
0f1700d38) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line
 735]
Event: 150.436 Thread 0x00007fd42971a800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x00000000f8486d90) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 152.113 Thread 0x00007fd42946d800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x00000000fad27750) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 152.439 Thread 0x00007fd42971a800 Exception <a 'java/net/SocketTimeoutException': Accept timed out> (0x00000000f8487388) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u144/9417/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 153.892 Thread 0x00007fd46c3bc800 Implicit null exception at 0x00007fd4984ca547 to 0x0000000000000000

[Víctor Almaraz · Accepted solution]

In case somebody is in the same situation:

After talking with Atlassian Support, fantastic service ;), we found that it was related with something wrong in our current database or bamboo home directory. It has nothing to do with the Bitbucket storage issue.

One test we make was changing Bamboo home directory and trying to reinstall from scratch, it works.

We have also rebooted the server to try to fix it.

In addition, (it may be related with the issue), they discovered that we have been infected, maybe when we had the old 5.0 version:

https://confluence.atlassian.com/bamboo/bamboo-security-advisory-2017-03-10-876857850.html

Many thanks to Atlassian Support team, that did a great job helping us to understand what was going on with our Bamboo Instance