Why has a Bamboo upgrade to 4.2 completely screwed my system?

[David FALLAS]

An earlier question (https://answers.atlassian.com/questions/81040/bamboo-4-2-upgrade-error-no-jvm-found) I had asked, indicated that installation of a more recent java runtime would resolve my problems with installing Bamboo 4.2.

Bamboo 4.2 installed OK after the java update, but now it appears that I cannot use my previous Bamboo data (SQLServer database) and worse still, it appears to have disabled access to FishEye where I now get an exception message:

Repository svnDHServices on http://kojira:8060/ failed: Error in remote call to 'FishEye' (http://kojira:8060) [AbstractRestCommand{path='/rest-service-fe/changeset-v1/listChangesets/', params={expand=changesets[-21:-1].revisions[0:29], comment=OLINTAAA-7, p4JobFixed=OLINTAAA-7, rep=svnDHServices}, methodType=GET}] : Connection refused: connect

Can anyone help me understand what may have happened here?

Can anyone assist me to recover to a point where we have a working system again?

This all seems to be far more difficult than it should be.

[Harry Chan · Accepted solution]

Hi, did you follow the steps in this guide? https://confluence.atlassian.com/display/BAMBOO/Bamboo+generic+upgrade+guide

Take note that your {BAMBOO_INSTALL} directory needs to be a new or empty directory. The legacy files need to be backed up and cleared away from the directory first.

[David FALLAS]

Hi Harry,

I believe that I did attempt to follow the steps, but I must've fallen foul of something along the way.

I have a database backup, and three separate directories which are also backed up. This was done prior to commencing the upgrade process.

C:\Users\administrator.SYSMEXNZ\bamboo-home (home)

C:\Program Files\Bamboo (install)

D:\Bamboo (working/data)

I have downloaded the installer for version 3.4, in the hope that I could restore the system to where it was. However I have failed at that too. I have attached the config file from bamboo-home. Perhaps something wrong there?

If you have any advice that might help me restore the system to a point where I can attempt the upgrade again, I would be very grateful.

(bamboo.cfg.xml.txt)

[David FALLAS]

In case it helps - I have attached the tail 100k or so of atlassian-bamboo.log (NOTE: this is a zipped file even though it has .log as file type - this page does not permit attaching .zip files).

It appears that there are a considerable number of missing or unresolvable elements.

BTW - this message from the log also leads to a dead link:

"For more details on how to fix this, see http://confluence.atlassian.com/x/1xy6D"

(atlassian-bamboo.log)

[David FALLAS]

..and a few more clues... these problems I am having all seem to be associated with installing the JDK.

The JDK is jdk1.7.0_06 from Oracle. Apart from that JDK directory, I also now have both jre6 and jre7 directories under the C:\Program Files\Java directory.

Bamboo v3.4 will work under certain conditions - when the JAVA_HOME environment variable is set to C:\Program Files\Java\jre7 - BUT - with this setting, the FishEye service refuses to start (wrapper.log contains the following)

"STATUS | wrapper | 2012/08/28 07:55:35 | --> Wrapper Started as Service

STATUS | wrapper | 2012/08/28 07:55:36 | Launching a JVM...

INFO | jvm 1 | 2012/08/28 07:55:36 | Error: missing `server' JVM at `C:\Program Files\Java\jre7\bin\server\jvm.dll'.

INFO | jvm 1 | 2012/08/28 07:55:36 | Please install or use the JRE or JDK that contains these missing components.

ERROR | wrapper | 2012/08/28 07:55:36 | JVM exited while loading the application."

In note that C:\Program Files\Java\jre7 does not contain a \server sub-directory - whereas \jre6 does

If JAVA_HOME is set back to jre6, the FishEye service WILL start, but still throws that exception when JIRA attempts to display source code check-ins.

It appears that the system I now have has some irreconcilable mismatches between the FishEye version (2.7.8), the Bamboo version (now reverted to 3.4) and Java (jre6, jre7 and jdk1.7.0)

[David FALLAS]

..and yet more...

I found a reference to the missing \server subdirectory - http://forums.atlassian.com/thread.jspa?threadID=32686. I followed the procedure there by manaully creating jre7\bin\server and copying into it the jvm.dll from the client sub-directory

Now the FishEye service starts (from Services applet), port 8060 is LISTENING, but I get this new exception when I try to navigate a browser to that port:

javax.servlet.jsp.JspException: javax.el.ELException: com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

javax.el.ELException: com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

You are running release CR2.7.8 FE2.7.8 (20111130052542 2011-11-30), please report your release number when reporting bugs.

[David FALLAS]

..and finally after finding some suggesitons at https://confluence.atlassian.com/display/FISHKB/'Timeout+exceeded+waiting+for+service'+Error+Message+After+Upgrade

I have attached the zipped wrapper.log after switching debug on (NOTE: once again this is a zip file, but I had to change the filetype suffix to .log

(wrapper.log.debug.zip.log)

[Harry Chan]

Hi, I went through the attached log and Fisheye has a problem authenticating to Crowd. Can you please check your Crowd integration first?

You can set JAVA_HOME to the where the JDK is and the JRE_HOME to where the JRE within the JDK is (there is a JRE folder inside the JDK).

Can you try to clear the Felix Cache at D:\ProgramData\Atlassian\FishEye\Data\cache\plugins\osgi-cache\felix? I've seen issues with that on upgrades/restarts when corrupts occur.

You might like to look for the cache in Bamboo and try clearing it too.

It does look like you have issues with your Java version. Best to un-install all the existing ones and stick with 1 version of Java.

[David FALLAS]

Thanks for the reply.

I am not sure what you mean about Crowd integration - we do not have Crowd installed.

I have uninstalled Java 7, (I tried to remove Java 6 - but the server warned me that some applications would not run if I did so, and left it).

I set the environment variables as advised..

JAVA_HOME=C:\Program Files\Java\jdk1.7.0_06

JRE_HOME=C:\Program Files\Java\jdk1.7.0_06\jre

I cleared everything out of the FishEye Felix sub-directory. I could not find a sister sub-directory by this name in Bamboo.

I rebooted the server.

The version of Bamboo is 3.4 (I removed version 4.2 and re-installed the version that was previously installed)

The version of FishEye is 2.7.8 (no upgrade to 2.8 has been attempted yet)

The system appears to be in even more of a mess now than it was earlier. Neither Bamboo nor FishEye services will start at all.

Please, I need some advice here on what to do next. I have a number of developers who want to get on with their work. Atlassian software has performed pretty well for the last 9 months up until this fiasco, but nevertheless it is still in a period of trial for many of our developers who do not yet trust it. Upgrading is beginning to look like rather a treacherous affair, and the software will be decommissioned if this cannot be resolved within a reasonable amount of time.

[Harry Chan]

INFO   | jvm 1    | 2012/08/28 10:01:57 | com.cenqua.fisheye.user.AuthenticationException: Problem communicating with Crowd

I saw the above in your logs. Do you know how Fisheye authenticates?

Can you please attach the logs for your new situation? It is best adviced to test upgrades in a UAT environment first to work out all the kinks. If you do need help with upgrades on a commercial basis, it's best to find an Atlassian consulting company that will do the upgrade for you.

[Harry Chan]

Make sure the java command runs from command line. Try running "java -version" and confirm it picks up the right version of java. I would check wrapper.conf in the conf folder to see if the settings are correct. Make sure there is enough memory on the servers. Are there any error logs on the Windows side of why the service isn't running?

[Harry Chan]

That definitely means Java hasn't been installed properly. You can add the java bin dir to your Windows path variable to fix it.

[David FALLAS]

I appear to be unable to run java -version from the command prompt - unless I CD to the Java\..\bin sub-directory - is this usual?

c:\Program Files\Java\jdk1.7.0_06\bin>java -version

java version "1.7.0_06"

Java(TM) SE Runtime Environment (build 1.7.0_06-b24)

Java HotSpot(TM) Client VM (build 23.2-b09, mixed mode, sharing)

FishEye wrapper.conf is attached

(fisheye.wrapper.conf.txt)

Bamboo wrapper.conf attached

(bamboo.wrapper.conf.txt)

Plenty of memory is available (2.04 cached, 600MB free from 4.0GB total)

Windows Event Viewer exceptions:

The Bamboo build server service terminated unexpectedly. It has done this 1 time(s).

The Fisheye service terminated unexpectedly. It has done this 2 time(s).

As for calling in an Atlassian consulting company, that would make the entire exercise somewhat pointless. We are not a large company - just one looking for best-of breed software to assist us do our jobs. JIRA, FishEye and Bamboo seemed to be what we were looking for.

[David FALLAS]

Thanks once again. We may be making a little bit of progress.

Bamboo and FishEye are now both reported in Windows Event Viewer to have started OK. I see an application listening on port 8085 - and on port 8060 - I assume Bamboo and FishEye respectively.

I can connect directly to the Bamboo URL and it's Dashboard displays - however the link between Bamboo and JIRA does not seem to function as JIRA asks me to recheck Bamboo configuration because it is not currently available.

I cannot get any response from the FishEye URL other than the message I posted earlier in this thread -

Your request was unable to be processed

javax.servlet.ServletException - javax.servlet.jsp.JspException: javax.el.ELException: com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

javax.servlet.jsp.JspException: javax.el.ELException: com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

javax.el.ELException: com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

com.atlassian.applinks.host.OsgiServiceProxyFactory$ServiceTimeoutExceeded: Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

Timeout exceeded waiting for service - com.atlassian.applinks.spi.link.MutatingApplicationLinkService

You are running release CR2.7.8 FE2.7.8 (20111130052542 2011-11-30), please report your release number when reporting bugs.

[Harry Chan]

Fisheye has duplicate/corrupt jars. Follow the link you posted earlier to fix it. That should solve it.

[Harry Chan]

Sorry, would like to give you a better response but i'm answering on my phone since you sound urgent. https://confluence.atlassian.com/display/FISHKB/'Timeout+exceeded+waiting+for+service'+Error+Message+After+Upgrade

[David FALLAS]

Hi again - if you are referring to:

"For more details on how to fix this, see http://confluence.atlassian.com/x/1xy6D"

That link is dead. Is there an updated link for that issue (Dr Google hasn't been able to find one)

[David FALLAS]

Harry, thanks again. I do appreciate the response.

I have checked for duplicate jars as per the link and cannot see any.

The additional parameters for java all seem to be present and correct in (FishEye) wrapper.conf

I have added the explicit %JAVA_HOME%/bin in wrapper.conf as suggested (although that bin sub-directory is now also in the windows path environment variable)

wrapper.java.command=%JAVA_HOME%/bin/java

There are no entries in <FISHEYE_INST>/var/plugins/config/pluginstate.properties

Added the additional property to wrapper.conf as suggested

wrapper.java.additional.14=-Djsse.enableCBCProtection=false

I am coming to the conclusion that the JDK that is now installed on our server is not compatible with FishEye 2.7.8. Should I just go ahead and install FishEye 2.8 regardless?

[Harry Chan]

Can you please repost your Fisheye log in this case - the latest one? I've gone through the old one and it's either a corrupted download or a corrupted plugin in your system somewhere since the Java version should already have been resolved.

[David FALLAS]

Seems my last comment was lost. As does it seem was yours. Nevermind, I'll try again.

I followed the suggestions at that link. I checked the various sub-directories for duplicate jars but there were none.

I checked the instance directory file plugin.properties, but that had no entries at all (other than a comment)

The java issue should be sorted as you say.

That entire last log from yesterday is attached.

[rocky janson]

Can you please repost your Fisheye log in this case - the latest one? I've gone through the old one and it's either a corrupted download or a corrupted plugin in your system somewhere since the Java version should already have been resolved.

hosting with dedicated ip

[David FALLAS]

Yay!!! - I can now open the FishEye site again. I installed the latest JDK to try to get the Bamboo 4.2 upgrade working. That installed itself in C:\Program Files\Java\jdk1.7.0_06 - so I guess it is Java 7.

Harry, a special thanks for all your patience and for eventually sleuthing out the reason.

Now all I have to do is figure out how to re-create all my Application Links!

[Harry Chan]

Did you upgrade from Java 6 to Java 7? (I think you mentioned you did) Please remove the INST/cache/plugins/osgi-cache directory. Explanation https://jira.atlassian.com/browse/FE-3916. I found this whilst going through your logs.

[David FALLAS]

Hi Harry, Thans for the advice. I can now see all comments - my apolgoies for not knowing why they were apparently 'missing'.

Perhaps the previous file was too big (620k) or the name was not acceptable. I'll try a rename with .log first - then if that doesn't work - I'll attempt to extract just the last time I tried to connect.

(wrapper.zip.log)

[Harry Chan]

Hi David, nothing got lost. If there are too many comments to a post it simply gets hidden. You have to press show all comments. Sorry, do you know which post you attached the logs to? I can't find it.

[David FALLAS]

I have made two posts as comments on the answer - however both have not been added. I note your last question was also lost from the thread

"Can you please repost your Fisheye log in this case - the latest one? I've gone through the old one and it's either a corrupted download or a corrupted plugin in your system somewhere since the Java version should already have been resolved."

I will attempt to add a new comment at this level to see if it remains in the thread

Harry - in answer to the last question, I have followed the suggestions from the link https://confluence.atlassian.com/display/FISHKB/'Timeout+exceeded+waiting+for+service'+Error+Message+After+Upgrade

1. I could not find any duplicate jars in the sub-directories specified

2. There were no entries in pluginstate.properties (apart from a comment)

3. I did not increase the JAR timeout setting - until we rule out other causes, this is just noise as the server is well specced and the timeout was never an issue before

4. As you observed, we believe the java issue has been resolved by adding the bin folder to the PATH environment variable.

I enclose the complete FishEye wrapper.log. It is zipped, but has a .txt file type (as before)