Bamboo Server Migration Approach & Issues

[Sneh Patel]

Context:

We are currently in the process of migrating our Bamboo server hosted on CentOS 6.10 to RHEL 8.9. Our current Bamboo version is v7.0.4. We want to upgrade it to v9.2.6 or a later LTS version after migrating our server. Both the database and the app live within the same server. After migration, we will like to have a separate database server. Also, Bamboo is connected to other Atlassian products like Bitbucket and Jira (uses Jira authentication as well). 

We referred to the following Atlassian documents:

• Moving Bamboo Between Machines 
• Cloning Bamboo Instance 

Moving Bamboo Between Machines document recommends to use the UI utility tool to export the Bamboo instance to the new server. However, upon trying that approach, it took too long and we can't afford to have our Bamboo instance unavailable for that long. 

Our Approach: 

By referring to Cloning Bamboo Instance : 

In order to test the migration and upgrades, we decided to clone our existing Bamboo instance (prod) and use it as a test instance. We will then try to migrate this test instance to a different server in order to test the migration. 

In order to stand the test instance in the cloned server we did the following:

1. Cloned our existing Bamboo server (prod).
2. Modified the following bamboo.cfg.xml fields within the clone:
   • "bamboo.jms.broker.client.uri" to point to cloned server
   • "license.string" to our development license 
     NOTE:
     - We tried to modify the "serverKey" property as well but then we ran into errors with regards to decryption of certain things in the database (mismatch of encryption/decryption). 
     - We didn't change the database connection string because we cloned the server so the existing bamboo database was also cloned (therefore, we still use localhost). 
3. Modified the following administration.xml fields within the clone:
   • "myBaseUrl" to point to cloned server
   • "myInstanceName" to "Cloned Bamboo Server" 
4. Start the service in from the installation directory: start-bamboo.sh (within the clone). 
5. Immediately configured app links to point to our test Bitbucket/Jira instances (including crowd server). 

Once we successfully stand up the cloned instance (still a CentOS server), we will look to migrate it over to a new RHEL 8 server by using rsync to copy both the installation directory and the bamboo-home directory.

When migrating the database to a database server we will use pg_dump utility to export our test Bamboo database (from the cloned server) to the new database server. 

We will then start up the Bamboo service in the new RHEL 8 server. If all works at this point, then we expect to have successfully migrated the Bamboo server. We will then perform upgrades as suggested in https://confluence.atlassian.com/bamboo/bamboo-upgrade-guide-720411366.html. 

Problems: 

1. When we initially started the service in the cloned instance, Bamboo was connected to our production Bitbucket service, and so it caused some issues where PR builds were getting sent to the cloned Bamboo service (causing hanging builds). In the Cloning Bamboo Instance document, there is no note of this potential issue and how to avoid it. 
2. We were able to register a Bamboo agent to in our cloned Bamboo service. However, it seems like auto-trigger is not working. We expect it to auto-trigger upon PR creation. We checked/noted the following things: 
   • We checked the health of our linked repositories by going to Admin Setting > System Information > Linked Bitbucket Server instances and it was green (connected properly). 
   • We noticed that the branch was getting created within the plan however, no build is getting queued for that branch when PR was created. 
   • We looked at the following thread:  https://community.atlassian.com/t5/Bamboo-questions/Bamboo-not-triggering-build-for-branches-when-code-is-pushed-to/qaq-p/1164364  but couldn't find anything in the log. 
   • In atlassian-bamboo.log, we notice that there are still references to the old ssh Bitbucket url. 
   • We couldn't find anything in particular within the atlassian-bamboo.log that would suggest a root cause of the issue. 
   • We made sure that the files within bamboo-home directory are all owned by the root user. 

Questions:

1. Is our approach correct? Or is there a better way? 
2. We are currently stuck at the part where we are trying to stand up the test Bamboo instance (after cloning existing Bamboo server) due to auto-trigger being broken. Is there a way solve this issue so that we can proceed with migrating the test instance to a new server? 
3. If possible, can someone from Atlassian provide clear steps to successfully migrating Bamboo server (like how there are clear steps when migrating Migrating Bitbucket Server)? Kind of getting confused since we are facing issues with auto-triggering of builds (this is just 1 of the issues, there could be several other issues). 

[Shashank Kumar · Accepted solution]

Hello Sneh,

Welcome to Atlassian community.

Looking at your approach I don't see any problems, the only thing I don't see is that you have not modified server.xml file in the cloned Bamboo Instance. At minimum you'll need to change the proxy URL which should point to test instance URL if you are using any.

Below are the two documents which you can refer for cloning a Bamboo Instance.

• https://confluence.atlassian.com/bamkb/how-to-migrate-a-bamboo-instance-with-a-large-number-of-artifacts-and-build-logs-1093015034.html

• Cloning Bamboo Instance 

I agree that when you start the cloned instance, it triggers the build automatically as it would be connected to other apps via app links. There is a page created to explain this in detail, you can refer How to start up Bamboo with build plans, deployment environments, branch detection and build status updates disabled 

For your second problem I guess when you have cloned the Bamboo Instance, the repositories are still pointing to the old bitbucket URL as now you have changed the app link to point to the new test bitbucket instance, there are two ways in which you can fix this.

1. Either relink those repositories again in the cloned Bamboo Instance pointing to the repositories in the test bitbucket Instance.

2. Update the URL in bulk, refer https://jira.atlassian.com/browse/BAM-18273 this for the SQL

Once this is done I believe trigger Issue should be solved.

If you looking for someone to have a close look at this and guide you better, I believe you can raise a Support ticket with Atlassian ( If you have a valid active license ) and someone from Support team will be able to assist you further.

Regards,

Shashank Kumar

**please don't forget to Accept the answer if your query was answered**

[Sneh Patel]

Hi @Shashank Kumar thank you for your response. In response to your suggestions: 

1. Yes we have already tried to relink. The repositories are pointing to the new app link. I even tried to clone a plan with the repo link pointing to our test Bitbucket. However, still no luck. 

2. Haven't tried this. I can try it out and see if that does the trick. 

Also, I don't think our server.xml uses proxy. 

But nonetheless, when we do get to production migration, do you approve of the approach outlined? Instead of using the UI utility to export our Bamboo instance, can we simply use rsync on the bamboo-home directory and the installation directory? That should still successfully migrate our instance right?  

[Shashank Kumar]

Hello Sneh,

For the Trigger problem you can try to change the URL and see if it works. Also try to add a fresh linked repository to a new test plan and see if this trigger works. This test will help you validate if the app link is working properly.

Yes lot of people don't use Import and Export UI utility as it takes a lot of time for import and Export and not suggest-able for large instances.

There are two basic things which you have to keep in mind

1. The first step for you is to clone the current PROD instance to another server - This should be on the same version on which PROD is running ( no upgrade yet here )

1.1 Clone home, install directory and copy it to new server.

1.2 Also clone the DB details and update bamboo.cfg.xml file in cloned bamboo-home directory with the cloned DB details, you can use the same DB but if you want to backup or rollback you'll have problems. 

1.3 Here you have 3 things ( cloned home, cloned install , cloned DB ) 

1.4 if the path of home directory is changing, make sure to update bamboo-init.properties file with new path. 

1.5 Now you can start the new instance, make sure the old PROD instance is not running

2. After the cloned instance is up and running you can plan for upgrade to another version.

This is just a very high level summary of what you should be doing, you can refer the above attached documents for all the detailed steps including backup.

Also please test this a test environment before trying into production.

Regards,

Shashank Kumar

**please don't forget to Accept the answer if your query was answered**

[Sneh Patel]

Thanks @Shashank Kumar, I just tried to copy over the home directory and installation directory to another test server (manually), and the trigger seemed to have worked for one PR in a specific repo. However, I get the following error when I created another PR in another repo in our test Stash and thus, does not queue the build: 

2024-02-29 15:33:56,569 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-7] [BranchCreatedEventListener] Received stash branch creation event [ base, sneh-test-pr2]
2024-02-29 15:34:01,498 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-16] [PullRequestAwareEventListener] Received pull request opened event [ base, from: refs/heads/sneh-test-pr2, to: refs/heads/master]
2024-02-29 15:34:01,649 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-16] [LightweightBranchCreationServiceImpl] Created branch with key CHEF-CHB127 for VCS branch VCS Branch [sneh-test-pr2]
2024-02-29 15:34:01,882 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-7] [NativeGitOperationHelper] Could not find [sneh-test-pr2] in remote refs.
2024-02-29 15:34:01,987 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-7] [NativeGitOperationHelper] Could not find [sneh-test-pr2] in remote refs.
java.lang.RuntimeException: com.atlassian.bamboo.plugins.stash.repository.InvalidStashRepositoryException: Git: Cannot determine head revision of 'ssh://git@<prod_stash_url>:7999/chef/base.git' on branch 'sneh-test-pr2'. Branch has probably been removed.
Caused by: com.atlassian.bamboo.plugins.stash.repository.InvalidStashRepositoryException: Git: Cannot determine head revision of 'ssh://<prod_stash_url>:7999/chef/base.git' on branch 'sneh-test-pr2'. Branch has probably been removed.
2024-02-29 15:34:01,987 INFO [6-DelayedChangeDetectionThread:pool-12-thread-64] [NativeGitOperationHelper] Could not find [sneh-test-pr2] in remote refs.
com.atlassian.bamboo.repository.RepositoryException: com.atlassian.bamboo.plugins.stash.repository.StashRepositoryException: com.atlassian.bamboo.repository.RepositoryException: command [/usr/bin/git log -p --name-only --encoding=UTF-8 --format=[<random_chars>_BAM_hash]%H%n[<random_chars>_BAM_author_name]%aN%n[d31bfa5_BAM_author_email]%ae%n[<random_chars>_BAM_timestamp]%ct%n[<random_chars>_BAM_commit_message]%s%n%b[<random_chars>_BAM_commit_message_end]%n[<random_chars>_BAM_file_list] -1 sneh-test-pr2 ^master] failed with code 128. Working directory was [/data/bamboo-home/xml-data/build-dir/_git-repositories-cache/<some_random_hash>]., stderr:

I have no idea why it is still using the prod stash url when it should be using the test Stash url since our test Bamboo is connected to test Stash. Also, when I try to create a PR in test Stash (for the repo which doesn't queue the build), I do not see any reviewers automatically loaded on, whereas I see reviewers automatically loaded for the repo which has no problem queuing builds. Not sure if the issue is related.

Could this be because of caching? 

[Shashank Kumar]

Hello Sneh,

This is because you have cloned everything from PROD into a test environment.

Just changing the application link will not change the URL of the already copied ( cloned ) repositories from Production.

There are two ways you can fix this.

1. Delete all the Repositories ( Linked , Project and Plan Repositories ) and them again, this will make sure these repositories are pointing to the test bitbucket instance and not Prod bitbucket instance.

2.You can also do this in bulk by running the SQL query which I shared earlier.

3. You can try to do a fresh Reindexing which is normally recommended when a prepare a new environment after cloning, refer Bamboo Reindexing 

4. For any repository if you see PROD URL in the logs you can check what URL it is pointing to by querying VCS_LOCATION table.

Regards,

Shashank Kumar

[Sneh Patel]

Hi @Shashank Kumar thank you for your suggestions. My apologies for taking too much of your time. Based on your suggested options here are my queries:  

1. From options 1-3 that you laid out, we can try any ONE of them to resolve the issue right? 
2. I tried option 1 - deleted 1 linked repo (for which auto-trigger was not working) and re-added it but no luck. Is option 2 the only option? Option 2 does not seem trivial and we have a lot of repositories and also, the xml_definition field is a composition of various tags that would require parsing. This may corrupt our cloned db (for test purposes).
3. Do you have to do option 3 regardless of whether you do 1 or 2? What exactly does re-indexing do? 

[Shashank Kumar]

Hello Sneh,

No need for apologies as we are all here to help each other on community :)

1) Coming to reindexing - This is not a 100% solution to the problem which you are seeing. Reindexing is advisable when you copy data from one DB to another and will help to eliminate any caching issue etc ( this may or may not be applicable for your problem )

The problem which you are seeing is that you have created a Bamboo test instance from Prod which means it contains all the Bitbucket server PROD URL for the repositories unless you fix it you'll not be able to trigger builds from Bitbucket Test Environment even if application link is pointing to Bitbucket test, this will work good if you link a new repositories.

The two options which I suggested above are the only thing I can think of.

Below are the update queries for postgres and MySQL DB as explained in https://jira.atlassian.com/browse/BAM-18273

PostgresDb and MySQL:

update VCS_LOCATION as V
   set xml_definition_data = replace(xml_definition_data,'<old-url>','<new-url>');

 
MSSQL Server: 

update  VCS_LOCATION  set XML_DEFINITION_DATA = replace( cast( XML_DEFINITION_DATA as nvarchar(max) ),'<old-url>','<new-url>'); 

You can try this option for 1 row and see if this works and then try in bulk.

Regards,

Shashank Kumar

[Sneh Patel]

I tried option 1. I removed the linked repo and re-added it again but no luck. So I guess option 2 (SQL queries) is the only option?

[Shashank Kumar]

In Option 1 after adding and removing the linked repository globally, you'll need to remove and re add this linked repository under the plans repository tab and resave the source code checkout task.

You'll also have to remove the git cache for this linked repository refer https://confluence.atlassian.com/bamkb/git-cache-removal-in-bamboo-329982070.html