Does Confluence Data Center have notification for status changes in Cluster?

[msilberman_arlo_com]

We have 3 nodes in our cluster and reliant on Load balancer to trigger notifications.

This only works when node /status fails.

Using a server side monitoring tool that reports on resources, but causes issues if we enable JVM monitoring.

We've noticed that in Cluster Monitoring the status of a node shows one offline when LB shows it as active.

Are there any plugins that support notifications for the Cluster status changes and or JVM resources?

[Tomas Karas]

Hi,

Based on some experience with Jira DC...

With focus on this part: "Cluster status changes and or JVM resources?"

The more plugins you add to an existing java application the more heavy it becomes.

Also say you might hit OoM or full codeCache, meaning the app and/or plugin in same java app will simply not respond at all(can't create thread, etc). How you want to address that situation?

If you want to check status of each node in cluster, you have few limited options:

a) configure your LB to also contain maintenance proxies. ie: configure it as usual but add extra ProxyPass(not balancers) with like /atlassian-node1 to first node, and so on. Assuming you are using cookie based balancing it would not matter if the tool would remove this prefix, needless to say though that one has to check the footer every now and then(for that I use stylus plugin and have browser side css that fixes version to bottom of my browser window, no scrolling required). As long as you are not logged off, you can even put it back in address bar, but it is not needed.

span#footer-build-information { bottom: 0px; position: fixed; color: darkred }

b) have/use extra http connector(conf/server.xml) that is configured for http based access as is standalone block that is not used for your LB.

b1) Connectivity directly to the port of the back-end is needed for this. Needless to say, this connection is not encrypted so your admin account password will fly over the network as is. Drawback, that could be addressed(ssh tunnel) or neglected depending on network situation and security. Of course one could put ssl cert/key in java keystore as an option for https.

b2) However if you want to tap into this with some "c)" monitoring script on localhost "b1)" will be non-issue.

c) external tools incl. monitoring and custom scripts. You will surely want this option in place because it can offer you so much more information about situation of each of your nodes, besides the usual cpu, ram, disk and is_it_responding? We had this, and I can say we are surely getting either the same thing back, or better.

c) ex1) script that analyses previous minute messages in log, looking for keywords 

c) ex2) script that automatically collects some information from node "b2)" webUI

c) ex3) script that takes threadDump or other sources(non webUI)

Then those metrics can be collected and plotted in graph.

Of course if there are known issues and how they manifest(stuckThread anyone? OoM, gc issues after which even catalina.out contains errors about it) then you can produce alerts right away from the box, as not everything is in ATSP...

I should probably mention, enable JMX for sure if it is not yet included in your version. I've had previously running jvisualvm connected to JMX on production endpoints, and when there was problem, we could easily look at live cpu and memory/gc graph with few hours of history. Another nice source of information that stops refreshing when tool dies. Likely that is not all that you can do over JMX...

Maybe your monitoring have plugin for JMX.

//Tomas