nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How can I add context to an alert?

James Allen

I'd like to be able to add context to an alert after it's created. For example, if there's an IP address in the alert I want to be able to automatically populate the alert with a geolookup of the IP that pulls back the country where it's located and put this into the alert. Ideally I'd be able to update the description with this new information, or add this as an extra property if not. If neither of those were possible we could probably also live with having this added as a comment to the alert.

This wouldn't need anything too complex, simply sending a GET request to a URL like:
https://ipinfo.io/18.165.242.47

Is it possible to trigger an API call like this and add the results to the alert?

Accepted answers

Tom Russell

@James Allen we do this on several alerts in our org; we call it "Alert Enrichment". We use the OEC capability to watch for inbound alerts with key labels on them. The OEC then runs scripts to pull extra information to add to the alert, before it's handled by a human. Some things that we do:

Host based alerts - Add a list of any other open alerts for that host
Disk space alerts - Hit the server and pull in a list the top disk-consuming directories, as well as any disk space increases that happened in the last 30 days.
CPU alerts - Hit the server and grab a list of top cpu-consuming processes

All of these let our associates do faster triage and, ultimately, resolve the issue quickly.

All comments

Tom Russell

Host based alerts - Add a list of any other open alerts for that host
Disk space alerts - Hit the server and pull in a list the top disk-consuming directories, as well as any disk space increases that happened in the last 30 days.
CPU alerts - Hit the server and grab a list of top cpu-consuming processes

All of these let our associates do faster triage and, ultimately, resolve the issue quickly.

James Allen

That sounds like the kind of thing we need, and I've been thinking of similar integrations too.

How do you use the OEC to update the alerts? Is that just over raw API calls or is there an integration back to the alert that called the OEC? I've struggled in the past to find much documentation on OEC, do you know of some good documentation anywhere?

Separate question, how have you presented this information within Opsgenie? I've built an HTML table for the description, but it's quite fiddly and would be complicated to have lots of different information in there. I've been creating dynamic hyperlinks to internal dashboards which show more information on the impacted host, but would be better if I could present data cleanly in Opsgenie itself.

Tom Russell

@James Allen we just call back to the API to make the changes. There are a couple of ways to do that:

Add a note - We don't do that as much because people miss them
Attach a file - Rarely, if ever
Append to the end of the description - this tends to be the most visible
- For short sections of content, we use this method the most. Opsgenie strips off any CSS, but does allow HTML content. This makes tables work, but they don't always look the way we want them to.
- Append a link to problem resolution documentation - We often link to documents in our Confluence space with detailed instructions.
- Append a link to dynamic dashboards - We do this a lot, adding links to Kibana or Dynatrace dashboards with information on the problem

We haven't found much documentation that's helpful, other than the Atlassian help pages.

James Allen

Thanks @Tom Russell , definitely sounds like the right approach for us too.

I discovered the pain of trying to build tables without CSS, not the best but with borders and non-breaking spaces ( ) it's been possible to get things to mostly look right.