Stateless Bamboo Agent deployment on Kubernetes with Helm and Flux

[zoltan_zvara]

Short problem description: Due to Bamboo Agent deployed on Kubernetes using Helm is inherently stateless, after a Pod crash or eviction, the Agent must be registered again manually.

Short question: how to enforce automatic registration of Bamboo Agents, when they have at least the same UUID?

Below is a WIP Agent Dockerfile:

FROM atlassian/bamboo-agent-base:6.10

COPY docker-authentication.json /home/bamboo/.docker/config.json
COPY bamboo-agent.cfg.xml /home/bamboo/bamboo-agent-home/bamboo-agent.cfg.xml

USER root

RUN chown ${BAMBOO_USER} /home/bamboo/bamboo-agent-home/bamboo-agent.cfg.xml
RUN chmod 755 /home/bamboo/bamboo-agent-home/bamboo-agent.cfg.xml

RUN apt-get update && \
    apt-get install maven -y && \
    apt-get install git -y

RUN apt-get install unzip -y
RUN apt-get install zip -y

RUN curl -s "https://get.sdkman.io" | bash
RUN /bin/bash -c "source /root/.sdkman/bin/sdkman-init.sh"

RUN bash -c "source /root/.sdkman/bin/sdkman-init.sh && \
   sdk list java && \
    yes | sdk install java 11.0.6.hs-adpt && \
    yes | sdk install maven 3.6.3 && \
    yes | sdk install sbt && \
    rm -rf /root/.sdkman/archives/* && \
    rm -rf /root/.sdkman/tmp/*"

RUN chmod 766 -R /root
RUN chown ${BAMBOO_USER} -R /root

USER ${BAMBOO_USER}
RUN ${BAMBOO_USER_HOME}/bamboo-update-capability.sh "system.builder.mvn3.Maven 3.3" /usr/share/maven
RUN ${BAMBOO_USER_HOME}/bamboo-update-capability.sh "system.git.executable" /usr/bin/git
RUN ${BAMBOO_USER_HOME}/bamboo-update-capability.sh "system.jdk.JDK 11 (AOJDK)" /root/.sdkman/candidates/java/current
RUN ${BAMBOO_USER_HOME}/bamboo-update-capability.sh "system.builder.sbt.SBT SDK" /root/.sdkman/candidates/sbt/current

The `bamboo-agent.cfg.xml` contains the following:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
    <buildWorkingDirectory>/home/bamboo/bamboo-agent-home/xml-data/build-dir</buildWorkingDirectory>
    <agentUuid>f4747352-3ffb-4e4a-b52e-fea06f68647a</agentUuid>
</configuration>

The above configuration file is the only one that survives restarts. This is intended. I don't want to persist the application files, even if those are bootstrap files downloaded from the main Bamboo server. 

When the Pod is restarted, the Bamboo main server would not accept `f4747352-3ffb-4e4a-b52e-fea06f68647a` UUID, even though it seems that the connection is made, stuff is downloaded, but "registration" could not be completed, since the Bamboo server thinks, this is a different Agent.

It seems to me so far that such stateless deployments (cloud native?) are not yet supported or I'm missing something.

Is there any way to make sure that the Bamboo main server accepts this "seemingly old" but restarted stateless Agent with the same UUID as before - even though there is a bootstrap phase for that Agent that must be taken care of?

[Marcin Gardias · Accepted solution]

Hi,

Yes, it is supported.

What I think happens though is that up restart, your agent connects from a different IP. When agent is first connected, the UUID is matched to _exact_ IP chain (IP of the agent host + IPs of the proxy servers in between). When you start a new container in Kube, it is likely to be assigned a different IP.

Here's what you do: you go to Agents -> Agents Authentication, find the UUID and manually edit the IP filter so that it matches all possible IPs  your agent can connect from.

[zoltan_zvara]

I see I have experimented with that as far as I can remember, but I will give it another run.

What I have now is the following:

*.*.*.* {UUID} Approved

Logs:

Apr 15, 2020 12:46:10 PM A remote agent is loading on atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffbndqm (172.101.0.16).
Apr 15, 2020 12:46:17 PM A remote agent is loading on atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffbndqm (172.101.0.16).
Apr 15, 2020 12:48:14 PM Remote agent "atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffbndqm" has registered.

I will report back once we restart the container.

[zoltan_zvara]

Now deleted the Pod, and upon restart & boot, the following error is printed on Agent Pod:

There is no new remote agent trying to register.

It is not allowed to register. I bet that the IP matching works. There seems to be something else to Agent's unique ID.

The agent then goes into a JVM exit-restart cycle:

ERROR | wrapper | 2020/04/15 13:27:14 | JVM exited unexpectedly.STATUS | wrapper | 2020/04/15 13:27:19 | JVM process is gone.STATUS | wrapper | 2020/04/15 13:27:19 | Launching a JVM...INFO | jvm 16 | 2020/04/15 13:27:19 | WrapperManager: Initializing...

After a few crash-restart cycles, the Agent is allowed to register. It seems that Bamboo is not using strictly the UUID, but also the agent's name. The previous one has to be TTLd by Bamboo and the new one not exactly detected as the same Agent.

Apr 15, 2020 1:28:54 PM A remote agent is loading on atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffpw82p (172.101.0.16).

Apr 15, 2020 1:28:56 PM Remote agent 'atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffbndqm' was unresponsive and has gone offline.

Apr 15, 2020 1:29:19 PM Remote agent "atlassian-bamboo-agent-master-atlassian-bamboo-agent-cb6ffpw82p" has registered.

[Yevhen]

What you can also do is write a little Python or Golang program that would save bamboo-agent.cfg.xml to a configmap, patch a statefulset or deployment and restart. Not ideal, but this way you will save the xml as a configmap and mount it as a file, so that it's persisted across restarts.

[zoltan_zvara]

May I have a confirmation on the fact that this is or it not supported?