Problem with builds not finishing required a build agent reboot

[Steve Messer]

For the past 2 or 3 weeks we have been seeing build issues.

In the Web UI it just says building slower than usually and hangs indefinitely.

When a build fails the logs show the following three entries:

  simple 22-Aug-2023 17:29:50 Finalising the build...  
  simple 22-Aug-2023 17:29:50 Stopping timer.
  simple 22-Aug-2023 17:29:50 Build UT-RPER347-JOB1-21 completed.

When a build succeeds the logs show the following after these three lines

simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'NCover Results Collector'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'JMeter Aggregator Build Processor Server'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'Build Hanging Detection Configuration'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'Bamboo project link creator'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'Cucumber JIRA updater'
simple 22-Aug-2023 17:29:56 Total issues created:0
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'Clover Delta Calculator'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'Maven Dependencies Postprocessor'
simple 22-Aug-2023 17:29:56 Running on server: post build plugin 'TestComplete Task JIRA updater'
simple 22-Aug-2023 17:29:56 All post build plugins have finished
simple 22-Aug-2023 17:29:56 Generating build results summary...
simple 22-Aug-2023 17:29:58 Saving build results to disk...
simple 22-Aug-2023 17:29:59 Store variable context...
simple 22-Aug-2023 17:29:59 Indexing build results...
simple 22-Aug-2023 17:29:59 Finished building UT-RPER347-JOB1-21.

We aren't sure what to look for. 

The log files are large (54MB) due to the test data that gets logged. Not sure if it is possible to clean that up.

Any suggestions would be appreciated. We typically get one or two builds per build agent reboot.

[Steve Messer]

Update: We found the following error in the build agent log:

java.lang.OutOfMemoryError: Java heap space

We doubled the heap size from 512 to 1024 and so far, so good.