Is my Bamboo Export Process still in progress ?

[yash_shanker24]

Host OS - Windows Server 2016
Bamboo Server Build -  60103
Database - Internal 

9 days ago, I started the Full System Export process [Settings > Overview > System > Export > Selected [-Export Results , - Export Artifacts, -Export Build Logs]] for my Bamboo Instance. Yes, it is an non maintained Bamboo Instance with data persisting from 5 years back.

Initially I was the getting the error/errorEvents.action loading on the web browser saying 

"Bamboo is currently exporting. Depending on the number of builds and test results, the time until Bamboo is available will vary. Bamboo will automatically become available as soon as this task is complete."

But now, it gives a time out, but I do see some activity happening in the logs. Below is what the logs read now:

2019-10-16 14:21:23,494 INFO [21-BAM::AirlineAgent::Agent:pool-38-thread-1] [BuildAgentControllerImpl] Agent 44859393 checking build queue for executables...
2019-10-16 14:20:00,922 INFO [23-BAM::CICDlocalAgent::Agent:pool-40-thread-1] [BuildQueueManagerImpl] takeBuildContext for agent 77758465 took 5569ms
2019-10-16 14:27:35,499 INFO [21-BAM::AirlineAgent::Agent:pool-38-thread-1] [BuildQueueManagerImpl] takeBuildContext for agent 44859393 took 2820ms
2019-10-16 14:25:03,010 INFO [20-BAM::TestElasticBamboo::Agent:pool-37-thread-1] [BuildAgentControllerImpl] Agent 32505857 checking build queue for executables...
2019-10-16 14:30:48,084 INFO [18-BAM::QWIK Agent::Agent:pool-35-thread-1] [BuildAgentControllerImpl] Agent 27197441 checking build queue for executables...
2019-10-16 14:29:37,065 INFO [19-BAM::TG Agent::Agent:pool-36-thread-1] [BuildAgentControllerImpl] Agent 28868609 checking build queue for executables...
2019-10-16 14:36:39,924 INFO [21-BAM::AirlineAgent::Agent:pool-38-thread-1] [BuildAgentControllerImpl] Agent 44859393 checking build queue for executables...

When I try to load Bamboo in  my web browser, no entry for that request is made in the logs.

Is the Export Process still in progress ?
Has it ever taken anyone this much time ?
What should my next steps be ?

[Daniel Santos · Accepted solution]

Hi @yash_shanker24

While the export is running you should see a pair or message like:

2019-10-17 14:41:40,064 INFO [http-nio-8085-exec-12] [XmlMigrator] Exporting with: com.atlassian.bamboo.buildqueue.migration.stream.AgentAssignmentMapper
2019-10-17 14:41:40,317 INFO [http-nio-8085-exec-12] [XmlMigrator] Finished writing to file: /Users/dsantos/atlassian/bamboo/6.10.2/home/exports/export_atlassianbamboo_61006_20191017.zip/db-export/agentAssignments.xml

The first one is the data class being exported and the second one is the export of that class represented in an XML file.

When the export finishes you should see a message like:

2019-10-17 14:41:41,357 INFO [http-nio-8085-exec-12] [XmlMigrator] Unmounted /Users/dsantos/atlassian/bamboo/6.10.2/home/exports/export_atlassianbamboo_61006_20191017.zip
2019-10-17 14:41:41,357 INFO [http-nio-8085-exec-12] [XmlMigrator] Export completed. 9.544 s

In order to move forward on this one, please help me with:

• What is your intention with this export? (migration, upgrade, etc)
• Do you intend to change the database flavor? (Postgres, SQL Server, Oracle or MySQL)
• Are you using this instance at the moment?
• Was the instance paused for this export?

This will help me think about what could be the best steps for moving forward.

[yash_shanker24]

Hi @Daniel Santos 

• What is your intention with this export? (migration, upgrade, etc)
  • Migration to an external DB and upgrade Bamboo version to 6.10.3.
• Do you intend to change the database flavor? (Postgres, SQL Server, Oracle or MySQL)
  • I intend to use MS SQL Server.
• Are you using this instance at the moment?
  • No.
• Was the instance paused for this export?
  • Yes, I paused it, and started the export process only when it was paused.

I got a similar log output as you have mentioned in the first part of your answer. But it is stuck at the zipping artifacts. See below.

2019-10-09 12:17:53,229 INFO [http-apr-443-exec-25] [XmlMigrator] Exporting with: com.atlassian.bamboo.migration.stream.IdGeneratorMapper
2019-10-09 12:17:53,229 INFO [http-apr-443-exec-25] [XmlMigrator] Finished writing to file: D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\db-export\idGeneratorData.xml
2019-10-09 12:17:53,557 INFO [http-apr-443-exec-25] [XmlMigrator] Writing xml to file: D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\db-export\planBranchPullRequests.xml
2019-10-09 12:17:53,557 INFO [http-apr-443-exec-25] [XmlMigrator] Exporting with: com.atlassian.bamboo.migration.stream.PlanBranchPullRequestMapper
2019-10-09 12:17:53,573 INFO [http-apr-443-exec-25] [XmlMigrator] Finished writing to file: D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\db-export\planBranchPullRequests.xml
2019-10-09 12:17:53,698 INFO [http-apr-443-exec-25] [XmlMigrator] Zipping config dir:D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\configuration
2019-10-09 12:17:54,012 INFO [http-apr-443-exec-25] [XmlMigrator] Zipping build dir:D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\builds
2019-10-09 12:17:54,012 INFO [http-apr-443-exec-25] [XmlMigrator] Zipping artifacts dir:D:\Bamboo\exports\export_atlassianbamboo_60103_20191009.zip\artifacts
2019-10-09 12:17:56,673 INFO [18-BAM::QWIK Agent::Agent:pool-35-thread-1] [BuildAgentControllerImpl] Agent 27197441 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [17-BAM::Default Agent::Agent:pool-34-thread-1] [BuildAgentControllerImpl] Agent 131073 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [22-BAM::CICDAgent::Agent:pool-39-thread-1] [BuildAgentControllerImpl] Agent 46333955 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [19-BAM::TG Agent::Agent:pool-36-thread-1] [BuildAgentControllerImpl] Agent 28868609 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [21-BAM::AirlineAgent::Agent:pool-38-thread-1] [BuildAgentControllerImpl] Agent 44859393 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [23-BAM::CICDlocalAgent::Agent:pool-40-thread-1] [BuildAgentControllerImpl] Agent 77758465 checking build queue for executables...
2019-10-09 12:17:56,673 INFO [20-BAM::TestElasticBamboo::Agent:pool-37-thread-1] [BuildAgentControllerImpl] Agent 32505857 checking build queue for executables...

And I also found this log snippet :

2019-10-13 05:51:40,463 INFO [18-BAM::QWIK Agent::Agent:pool-35-thread-1] [BuildQueueManagerImpl] takeBuildContext for agent 27197441 took 45599ms
2019-10-13 05:53:07,216 INFO [19-BAM::TG Agent::Agent:pool-36-thread-1] [BuildAgentControllerImpl] Agent 28868609 checking build queue for executables...
2019-10-13 05:53:03,899 INFO [http-apr-443-exec-25] [XmlMigrator] Ensuring the the file has been unmounted
2019-10-13 05:53:03,899 INFO [23-BAM::CICDlocalAgent::Agent:pool-40-thread-1] [BuildAgentControllerImpl] Agent 77758465 checking build queue for executables...
2019-10-13 05:54:26,970 INFO [ActiveMQ Journal Checkpoint Worker] [DefaultIOExceptionHandler] Ignoring no space left exception, java.io.IOException: Java heap space
java.io.IOException: Java heap space
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)
at org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:433)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2019-10-13 05:58:08,940 INFO [20-BAM::TestElasticBamboo::Agent:pool-37-thread-1] [BuildAgentControllerImpl] Agent 32505857 checking build queue for executables...
2019-10-13 05:59:01,966 INFO [19-BAM::TG Agent::Agent:pool-36-thread-1] [BuildAgentControllerImpl] Agent 28868609 checking build queue for executables...
2019-10-13 05:59:04,785 INFO [23-BAM::CICDlocalAgent::Agent:pool-40-thread-1] [BuildAgentControllerImpl] Agent 77758465 checking build queue for executables...
2019-10-13 05:58:56,430 INFO [18-BAM::QWIK Agent::Agent:pool-35-thread-1] [BuildAgentControllerImpl] Agent 27197441 checking build queue for executables...
2019-10-13 05:58:14,538 INFO [21-BAM::AirlineAgent::Agent:pool-38-thread-1] [BuildAgentControllerImpl] Agent 44859393 checking build queue for executables...

I would also like to highlight here that my Bamboo instance [t3.2xlarge aws instance] is 5 years old and the Bamboo home directory is over 500GB in size, and I have around 1.3 TB  free space available on my drive. 

[Daniel Santos]

Hi @yash_shanker24

I think I get it now, you have an old instance (as stated before), but this instance is running with an embedded DB and now you need to generate an export to import it in a fresh instance and use an external DB. Please correct me If I'm wrong.

You don't need an export with artifacts, it can be done without it.

This would be my steps to approach this problem. I'm considering a migration to another server with external DB end an upgrade:

PART 1 - Migrate the server and DB

1. Backup
   1. Backup your Bamboo home directory (it will include DB, artifacts etc), assuming there is no custom path set for artifacts, etc
   2. Backup the installation directory. The intention here is to gather any tomcat configuration or application customization.
2. Reduce the instance size
   1. Try to clean old results that are not needed anymore. You can configure a more restricted global (and plan overriden global expiry) to reduce the build and deployment results. This will reduce the instance size.
   2. Try to remove projects that are not used.
3. Export the instance using only the Export results option
   
   [x] Export Results
   [  ] Export Artifacts
   [  ] Export Build Logs
   
   ⚠️ The Export Results option is what links the Artifacts and Build logs migrated via filesystem with the next installation.
   
   
4. Install the new instance with the same version
   1. This installation needs to consider using the same paths as the old one. This will make things simple.
   2. It needs to use the same Bamboo version. This is a must considering that part of the files will be migrated manually. Different versions of Bambo may have different folder structures to save files in the filesystem.
5. Migrate Bamboo Installation customizations/configurations
   1. Tomcat configurations
   2. Instance customizations (if you have them) 
6. Import the backup
   ⚠️Please consider that this process is even slower than the export, unfortunately.
7. Stop the instance
8. Copy back the bamboo artifacts and build logs
9. Start the instance and check the results

PART 2 - Upgrade the instance

1. Follow our Bamboo upgrade guide

Let's try to sync the strategy first and then we can get back to the export/import issues. What do you think about it?
Let me know if you have any questions about what I just shared above.

[yash_shanker24]

Hi @Daniel Santos  , I am happy to update you that my export and import finished on Monday this week and the upgraded Bamboo is functional. There were a couple of ups and down, but were solved with the help of a couple of support tickets and the Knowledge Base articles. Thanks.

[Daniel Santos]

Hey @yash_shanker24

thank you very much for coming here to update this thread =]

I'm glad that you were able to upgrade Bamboo and It's working as expected now.

Have a good one!