Unable to start Bitbucket 5.6.3 after upgrade from 5.1.0

[Eric Wittle]

I've attempted to upgrade Bitbucket 5.1.0 to 5.6.3. When trying to start Bitbucket server after the upgrade, I get the following exception stack trace in the server log. The configuration is very slightly customized to run SSL, and is running in a container off of an NFS persistent volume. The same persistent volume and container work fine with 5.1.0. 

My research by Google search for java.util.zip.ZipException indicate a potential corrupt jar file. However, I downloaded 5.6.3 multiple times, and I tried multiple methods to get it onto the NFS share with no luck. The download I used was in the .tar.gz format.

The JDK version is Oracle's JDK 1.8.0_161. Suggestions as to where to look next for the problem would be appreciated.

Bundled Elasticsearch started successfully

 

The current open files limit is set to less than 4096

Attempting to increase limit...

 Limit increased to 4096 open files

 

Starting Bitbucket webapp at http://localhost:7990

Exception in thread "main" java.util.zip.ZipException: error in opening zip file

 at java.util.zip.ZipFile.open(Native Method)

 at java.util.zip.ZipFile.<init>(ZipFile.java:225)

 at java.util.zip.ZipFile.<init>(ZipFile.java:155)

 at java.util.jar.JarFile.<init>(JarFile.java:166)

 at java.util.jar.JarFile.<init>(JarFile.java:130)

 at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:112)

 at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:106)

 at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:92)

 at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:83)

 at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:61)

 at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:57)

 at org.springframework.boot.loader.archive.ExplodedArchive.getNestedArchive(ExplodedArchive.java:123)

 at org.springframework.boot.loader.archive.ExplodedArchive.getNestedArchives(ExplodedArchive.java:110)

 at org.springframework.boot.loader.ExecutableArchiveLauncher.getClassPathArchives(ExecutableArchiveLauncher.java:72)

 at com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher.start(BitbucketServerLauncher.java:150)

 at com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher.main(BitbucketServerLauncher.java:99)

[Eric Wittle]

It was a permission issue, but a really odd one (to me). The normal file permissions were fine, but since I untarred the bitbucket distribution on the NFS share I'm running it from from my MacBook, there were extended attributes on the file, and somehow this screwed things up. When unzipped the bitbucket distribution from a Linux host onto the same NFS share, I was able to complete the upgrade.

Since I used the same user account (an LDAP user) from both clients, this was unexpected on my part.

If anyone else is curious whether they're seeing the same problem, look for the "@" sign at the end of the normal ls -l output on OSX. Here are the permission for the 5.9.1 directory that was failing, compared to the 5.1.0 one that worked.

Eric:5.9.1 eric$ ls -al app

total 72

drwxr-xr-x  1 eric  staff    296 Apr  7 11:49 .

drwxr-xr-x  1 eric  staff    330 Apr  7 12:19 ..

drwxr-xr-x@ 1 eric  staff    264 Apr  3 00:56 META-INF

drwxr-xr-x@ 1 eric  staff    264 Apr  7 11:43 WEB-INF

drwxr-xr-x@ 1 eric  staff    264 Apr  3 00:58 com

-rw-r--r--@ 1 eric  staff   1671 Apr  7 11:47 fatal.jsp

drwxr-xr-x@ 1 eric  staff    264 Apr  3 00:56 org

-rw-r--r--@ 1 eric  staff  28070 Apr  7 11:39 startup.jsp

drwxr-xr-x@ 1 eric  staff    568 Apr  3 00:56 static

-rw-r--r--@ 1 eric  staff    182 Apr  7 11:49 web-fragment-placeholder.config

Eric:5.9.1 eric$ ls -al ../5.1.0/app

total 72

drwxr-xr-x  1 eric  staff    296 Jan 27 08:47 .

drwxr-xr-x  1 eric  staff    364 Feb 10 15:54 ..

drwxr-xr-x  1 eric  staff    264 Jan 27 08:46 META-INF

drwxr-xr-x  1 eric  staff    264 Jan 27 08:47 WEB-INF

drwxr-xr-x  1 eric  staff    264 Jan 27 08:46 com

-rw-r--r--  1 eric  staff   1671 Jan 27 08:46 fatal.jsp

drwxr-xr-x  1 eric  staff    264 Jan 27 08:46 org

-rw-r--r--  1 eric  staff  28070 Jan 27 08:46 startup.jsp

drwxr-xr-x  1 eric  staff    568 Jan 27 08:47 static

-rw-r--r--  1 eric  staff    182 Jan 27 08:47 web-fragment-placeholder.config

Here's hoping others don't hit this.

[Danyal Iqbal]

The jar archive is corrupt/insufficeint permissions etc. Check user permissions!! (the one running bitbucket). Explore the contents of the jar archive with 7zip or something similar.

[Eric Wittle]

I saw that answer when I google searched the exception. I'm not at all confident that is the issue in my case. I downloaded the application twice, and copied it onto the NFS share three times. If the Jar is corrupt, it seems most likely that is corrupt at the source. I would be happy to explore the jar with a zip program, but the specific jar that this message relates to seems to not be mentioned anywhere I can see in the stack trace, so I don't know which one to check.

[Christian Glockner]

Hi Eric,

Can you post the full stack trace, including all "caused by" sections?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Christian Glockner]

Never mind, I just saw your update further up there :-)

[Eric Wittle]

Glad you saw the update, any additional suggestions to offer to triage the issue? I'm still running 5.1.0 at the moment, and would prefer to get this upgraded.

[Eric Wittle]

Trying again with bitbucket 5.9.1, and still seeing the exact same error stack when trying to start the container.

[Christian Glockner]

Hi Eric,

Somewhere a few lines below the exception there should be a section starting with "caused by" - this is the underlying exception that occurred - what does it say?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Eric Wittle]

There is no additional information below the stack trace provided, or I would have provided it. The only additional information in the log is a message that the Elasticsearch PID file is not found, then the log ends. Here's the last few lines of the log:

 at com.atlassian.bitbucket.internal.launcher.BitbucketServerLauncher.main(BitbucketServerLauncher.java:99)

 

No PID file was found for bundled Elasticsearch

Eric:bin eric$