We're using JSM and OpsGenie - should automated, initial incidents come into JSM or OpsGenie?

[Nathan Given]

We have multiple systems that send out "alerts" when there are failures (high CPU usage, hard drive running out of space, running out of memory, website down, application unhealthy, etc, etc).

Should we have these systems create incidents in JSM first, and then if they are critical enough, have an alert created in OpsGenie?

Or is it better to create alerts in OpsGenie and then auto-generate incidents in JSM?

I'm leaning towards the first option (External System > JSM > (if needed) Opsgenie), however, I've been told that External System > OpsGenie > JSM is the "preferred" approach.

While I try to figure out what "preferred approach" means from the person who told me, I figured I'd ask what everyone else is doing and the pros/cons they've seen of either approach.

How have you set things up?  Why did you set it up that way vs the other way?

Thank you~,

Nathan

[Andrew Laden]

It depends on a few things.

Opsgenie is very good at "pre-processing" alerts. Doing things like

• "de-duplicating" so multiple messages about the same item dont create multiple tickets, 
• Matching alert resolved to the open alert, so that you can auto-close things
• Categorizing alerts

So if your alert source is clean, and you dont need those things, then you could consider JSM to OpsGenie.

There are really 2 separate use cases for OpsGenie/Jira integration

1: Handling alerts from various monitoring tools. In this case, You feed the alert to OpsGenie first, let it preprocess (and optionally notify on call) about an issue, and then create a jira ticket to track the the issue. (perhaps using data you parsed out of the alert in opsgenie to set custom fields, etc) This leverages opsgenie's strengths in pre-processing and its integrations.

2: Handling issues raised in service desk. In this case, someone (or thing) opens a ticket in service desk. Service Desk tells opsgenie to open a alert, so that it can notify the oncall users of an issue. You can then manage the incident in JSM, and leverage OpsGenie for notification only. This is great for a helpdesk type situation, where a user can raise a "Critical" issue, and you can quickly page out to people and swarm on the issue.

The 2 are really different use cases and there are lots of gotchas if you try to do both at the same time. You need to decide how you want to leverage them. 

From what you described, I would have OpsGenie preprocess the alert and create the jira ticket. You will get much more out of opsgenie that way (and create many less jira tickets)

[Nathan Given]

Thank you, @Andrew Laden !

Another thing I found: The source system we have has the tendency to use "unsafe" characters when sending alerts (",',\, etc).

I've noticed that the Opsgenie parser has an "advanced mode" that let's you use µ#µ and §#§ as delimiters (insetad of = and &).

This definitely makes it easy to send unclean/dirty data to opsgenie and let it handle the parsing/preprocessing.

[Nathan Given]

PS: It turns out we may have to use both use cases anyway:

For automated alerts, we can send notifications to Opsgenie and then as needed, opsgenie will create JSM tickets.

But we also have the ability for humans to create high priority/critical incidents in JSM, and we definitely want to leverage the existing paging/notification capabilities of Opsgenie to notify the correct team and escalate if they don't respond.