Integrating Alerts with Logic Monitor

[#1]

TLDR: In the Logic Monitor integration with Operations, replace ##ALERTID## with ##INTERNALID## in the HTTP delivery alert data from Logic Monitor (provided by alertData.txt)

This isn't a question, but I wanted to post this here in case it helps someone with the same issue. We noticed inconsistencies in interactions between Logic Monitor and Jira Operations using the built-in integration in Ops. Acknowledgements from Ops to LM would intermittently fail and we weren't seeing all our LM alerts generated in Ops. 

We use the ingest API in Logic Monitor to generate alerts from scripts. These all come from the same host and data source, so even though they are represented as separate alerts in Logic Monitor, they often share the same alert ID. Logic Monitor notes in their docs that the ##ALERTID## token is not unique - it's used for deduplication of related alerts on their end. The preferred unique ID is the ##INTERNALID## token. This is also the only value that will consistently ack alerts.

Since the integration's default configuration uses an alias smart value of {{alertId}}, LM alerts sharing the same ##ALERTID## value will only generate a single alert in Ops, and attempting to ack the ALERTID value from Ops will fail. 

To fix it, we modified the alertData.txt template provided in the configuration by substituting the ##ALERTID## value with ##INTERNALID## and reverted the alias fields back to {{alertId}}. Now we are consistently receiving identical alerts in Ops from LM, and acknowledging on either side will properly ack the alert in both. 

Note: We thought we initially resolved it by adding {{_payload.internalId}} as a custom alert property and using that in the alias field for the create, close, and acknowledge actions. This resulted in 1:1 alert generation in LM and Ops; however, we found that when acknowledging multiple alerts that shared the same alert ID in Logic Monitor, only one alert would be acknowledged on the Ops side. It turns out that the acknowledge action ignores the alias in the integration config and just defaults to the alert ID value when linking to alerts in Ops.

[Tomislav Tobijas]

Oh, interesting @Ryan! Thanks for sharing 🙌

This just gave me one idea for implementing LogicMonitor or a similar tool in a couple of our environments. I'm curious - you said you implemented this for scripts... Have you considered using the same/similar thing for automations? That's something I was thinking while I was reading this post. 👀

P.S. I'm going to move this under the JSM Q&A section as Ops are more related to that. 

[Ryan]

@Tomislav Tobijas , are you referring to Jira Automations or Automation rules in Ops? If so, are you considering routing errors in those to alerts? I haven't considered that. 

Logic Monitor is our primary monitoring/alerting tool, but since we operate in Jira for service requests & incidents, we figured we might as well route alerts there as well. Logic Monitor has an HTTP Ingest endpoint that allows us to generate alerts via HTTP requests. We wrote custom log modules for PowerShell and python that automatically route errors to Logic Monitor for alert generation. These now also flow into Jira Operations so we can respond, escalate, document or generate incidents there.