Tuning AMQ for Large Bamboo Instances

[EddieW]

The default AMQ config seems to go against apache's advice for default, and scaled AMQ config.

http://activemq.apache.org/producer-flow-control.html#ProducerFlowControl-ConfigureClient-SideExceptions

But provided config disables flow control.

<property name="destinationPolicy">
        <amq:policyMap>
          <amq:policyEntries>
            <amq:policyEntry queue=">" producerFlowControl="false">
              <amq:deadLetterStrategy>
                <amq:sharedDeadLetterStrategy processExpired="false"/>  <!-- discard expired messages: -->
              </amq:deadLetterStrategy>
            </amq:policyEntry>
          </amq:policyEntries>
        </amq:policyMap>
      </property>
      <property name="systemUsage">
        <amq:systemUsage>
          <amq:memoryUsage>
            <amq:memoryUsage limit="128 mb"/>
          </amq:memoryUsage>
          <amq:storeUsage>
            <amq:storeUsage limit="100 gb"/>
          </amq:storeUsage>
          <amq:tempUsage>
            <amq:tempUsage limit="50 gb"/>
          </amq:tempUsage>
        </amq:systemUsage>
      </property>

We think this is causing a roque plan to essentially DoS the bamboo master server by overloading AMQ channels.

942754:2015-12-28 12:59:23,259 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 6275ms, which may be too long.
942755:2015-12-28 12:59:23,260 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@5832ad0a
942756:2015-12-28 12:59:23,262 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 422% 540 MB

Anyone have experience tuning more memory and more importantly enabling flow control?

[Przemek Bruski]

I wouldn't recommend enabling producer flow control until you upgrade to 5.10. With 5.10, it should be OK.

I don't recommend ignoring messages in buildTailQueue at all. This will have bad consequences.

[Przemek Bruski]

Note, may still be OK with flow control pre-5.10.

[EddieW]

Hey thanks @Przemyslaw Bruski ! Understand the concerns on ignoring messages, but it seems other queues like heartbeat, events etc are MORE important then build logs. But after playing I do see that buildTailQueue seems to handle more then just logs ("*and the process result message to the server*"). So the basic question is what configuration is most ideal if we know certain roque plans will bombard the master with too much verbosity. We have a 100 agent license but still can't go over ~25 without crippling the master. {code}<!-- Queue to send commands to the agent. Server => Agent --> <amq:queue id="agentCommandQueue" physicalName="com.atlassian.bamboo.agentCommandQueue"/> <!-- Queue to send events to the agent. Server => Agent --> <amq:topic id="remoteEventsTopic" physicalName="VirtualTopic.com.atlassian.bamboo.remoteEventsTopic" /> <!-- Queue to communicate back heartbeatQueue Messages back to server. Agent => Server --> <amq:queue id="heartbeatQueue" physicalName="com.atlassian.bamboo.heartbeatQueue" /> <!-- Queue to communicate back BambooMessages back to server. Agent => Server --> <amq:queue id="serverQueue" physicalName="com.atlassian.bamboo.serverQueue" /> <!-- Queue to send log messages and the process result message to the server. Agent => Server --> <amq:queue id="buildTailQueue" physicalName="com.atlassian.bamboo.buildTailQueue"/> <!-- Queue for agents to register itself. Agent => Server -{code}

[Przemek Bruski]

You will have gray builds with that config (due to dropped messages). How much log output are we talking about?

[EddieW]

Yes, exactly. But some incomplete build results still seems to be better then all agents going offline and UI locking up do to memory being used by ActiveMQ. Our last outage was caused by some logs with 1MM plus lines, some jobs generated 24,000,000 lines of output!! (yes, clearly those teams were coached on how to be less verbose, but I'd like the system to be able to defined itself better) Looking at latest ActiveMQ i wondered is LevelDB would be more performant then Khana? ALternately if build log messages could be isolated from the process results.

[EddieW]

(we do have a premier ticket open PS-3031 for this. We tried 5.9.7 in our lab and still brought it to its knees with 50 agents) I'll readily knowledge it's an extreme use case for so many logs, but Bamboo is mission critical for us, and we aren't comfortable with the risk of a few bad plans taking it down.

[Przemek Bruski]

I understand. Let's continue the discussion in the premier ticket.

[roy_lyons]

This is exactly the thread I have been looking for all my life....

I have several large instances of bamboo...  and the remote agents die from communication loss all the time!

I would love to know if there is a document anywhere describing some tuning that we can do for activemq.  I have angry QA testers who get mad that the whole instance locks up overnight during their regression and cucumber testing and they lose everything...

[EddieW]

Here's what seems to be holding up in our lab.

We configure the buildTailQueue with a distinct policy to ignore, and reject messages when memory limit is hit.  This allows other queues (like heartbears, registration, build events, etc) to be processed while some logging statements from remote agents are ignored.

<bean id="broker" class="com.atlassian.bamboo.amq.BambooBrokerService"
          init-method="start" destroy-method="stop">
      <property name="useJmx" value="${bamboo.broker.useJmx}" />
      <property name="brokerName" value="bamboo" />
      <property name="deleteAllMessagesOnStartup" value="${bamboo.ignore.server.state.on.restart}" />
      <property name="persistent" value="true" />
      <property name="dataDirectory" value="${bamboo.home}/jms-store" />
      <!--<property name="storeOpenWireVersion" value="10"/>-->
      <property name="destinationPolicy">
        <amq:policyMap>
          <amq:policyEntries>
            <amq:policyEntry queue=">" producerFlowControl="false">
              <amq:deadLetterStrategy>
                <amq:sharedDeadLetterStrategy processExpired="false"/>  <!-- discard expired messages: -->
              </amq:deadLetterStrategy>
            </amq:policyEntry>
            <!-- enabled flow contol on build logs to ensure registratino and build events are handled -->
            <!-- also sets memory to 1/2 total brokewe memory limit -->
            <amq:policyEntry queue="com.atlassian.bamboo.buildTailQueue" producerFlowControl="true" memoryLimit="128 mb">
              <amq:deadLetterStrategy>
                <amq:sharedDeadLetterStrategy processExpired="false"/>  <!-- discard expired messages: -->
              </amq:deadLetterStrategy>
            </amq:policyEntry>
          </amq:policyEntries>
        </amq:policyMap>
      </property>
      <property name="systemUsage">
        <amq:systemUsage sendFailIfNoSpace="true">
          <amq:memoryUsage>
            <amq:memoryUsage limit="256 mb"/>
          </amq:memoryUsage>
          <amq:storeUsage>
            <amq:storeUsage limit="100 gb"/>
          </amq:storeUsage>
          <amq:tempUsage>
            <amq:tempUsage limit="50 gb"/>
          </amq:tempUsage>
        </amq:systemUsage>
      </property>
    </bean>

Before

2015-12-30 11:51:57,918 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 341% 437 MB
2015-12-30 11:52:07,516 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 9598ms, which may be too long.
2015-12-30 11:52:07,516 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@28ce257
2015-12-30 11:52:07,537 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 337% 431 MB
2015-12-30 11:52:33,792 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 329% 422 MB
2015-12-30 11:52:47,971 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 14178ms, which may be too long.
2015-12-30 11:52:47,971 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@6e9cb32
2015-12-30 11:52:47,971 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 321% 411 MB
2015-12-30 11:53:15,466 WARN [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 27495ms, which is very long, this is a risk to system stability
2015-12-30 11:53:15,466 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@6cd08d49
2015-12-30 11:53:15,466 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 312% 400 MB
2015-12-30 11:53:22,429 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 6963ms, which may be too long.
2015-12-30 11:53:22,429 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@5c48aae7
2015-12-30 11:53:22,429 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 305% 391 MB
2015-12-30 11:53:25,188 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 2758ms, which may be too long.
2015-12-30 11:53:25,188 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@20d9c313
2015-12-30 11:53:28,079 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 290% 372 MB
2015-12-30 11:53:36,251 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 8172ms, which may be too long.
2015-12-30 11:53:36,251 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@582adc9b
2015-12-30 11:53:36,251 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 282% 362 MB
2015-12-30 11:53:46,175 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 9924ms, which may be too long.
2015-12-30 11:53:46,175 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@6578d306
2015-12-30 11:53:46,176 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 273% 350 MB
2015-12-30 11:53:50,185 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 4009ms, which may be too long.
2015-12-30 11:53:50,185 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@37eeb792
2015-12-30 11:53:53,482 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 3296ms, which may be too long.
2015-12-30 11:53:53,482 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@474ba468
2015-12-30 11:53:53,482 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 257% 330 MB
2015-12-30 11:53:59,885 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 225% 288 MB

After

2015-12-30 12:34:36,973 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 70% 180 MB
2015-12-30 12:34:42,674 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 5701ms, which may be too long.
2015-12-30 12:34:42,674 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@5e7b9c16
2015-12-30 12:34:47,811 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 110% 282 MB
2015-12-30 12:34:53,714 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 5903ms, which may be too long.
2015-12-30 12:34:53,714 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@4c7593bf
2015-12-30 12:34:53,716 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 107% 274 MB
2015-12-30 12:34:55,979 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 2263ms, which may be too long.
2015-12-30 12:34:55,979 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@52245a66
2015-12-30 12:35:05,149 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 9144ms, which may be too long.
2015-12-30 12:35:05,149 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@243f86cd
2015-12-30 12:35:05,149 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 101% 259 MB
2015-12-30 12:35:32,921 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 97% 249 MB

2015-12-30 12:35:56,836 WARN [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 23915ms, which is very long, this is a risk to system stability
2015-12-30 12:35:56,836 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@4e7fb00a
2015-12-30 12:35:57,645 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 93% 239 MB
2015-12-30 12:36:00,754 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 3109ms, which may be too long.
2015-12-30 12:36:00,754 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@27903fdd
2015-12-30 12:36:28,640 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Memory Usage: 89% 230 MB
2015-12-30 12:36:39,909 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] Processing of message took 11269ms, which may be too long.
2015-12-30 12:36:39,909 INFO [buildTailMessageListenerConnector-1] [BambooAgentMessageListener] The message is of type LogMessageStream, content com.atlassian.bamboo.v2.build.agent.messages.LogMessageStream@7939d289

You can see once memory goes over 100%, ignoring the log lines allows the server to quickly recover space.

But the UI is just slower, vs being unresponsive, and  agents do seem to stay online.

[EddieW]

Looking at http://activemq.apache.org/per-destination-policies.html I dont know enough to test anymore..