Integration with Grafana

[Susannah Roush]

Hi all,

We have many machines highly configured with Grafana. We found the Grafana-JSM integration and were, at first, delighted to use it to alert us when quick action is needed. We have over 15 Grafana-JSM integrations set up, each with different API keys. Let's say Machines A, B, & C, each with alerts for Problems 1, 2, & 3. 

Preferred behaviour would be that Machine A alerting with problem 2 would be a separate alert from Machine B alerting with problem 2. Right now, JSM puts them together, as part of deduplication. But then we miss that Machine B is having the same issue.

(There is a work around, but it requires modifying each incoming alert rule to add a prefix to the incoming alias.)

JSM uses the Alias provided by Grafana for identification of the alert. But why? When they are separate APIs?

Why do incoming alerts for different APIs (therefore different machines) get lumped together?

[Gabriela]

Hi Susannah, that's working as designed rather than a bug. JSM's Opsgenie engine dedupes purely on the alias of any open alert, account-wide, so the source integration or API key never factors into the match. To keep your machines separate, open the Grafana integration's Advanced settings and edit the Alias field to append a per-host value (drag the instance/host dynamic field in) so each box gets its own alias. One thing that catches people: mirror that same alias on the Close action too, otherwise Grafana's auto-close stops correlating. Field list is in Atlassian's Grafana integration docs.

[Susannah Roush]

Oh good catch on the Close action. Had not done that.

[Viswanathan Ramachandran]

hi @Susannah Roush good post. 

These alerts are based on the alias, not the integration/API key. The API key only determines which integration receives the alert. If you need alerts from different machines (or logical resources) to remain separate, include a stable unique identifier in the alias such as the application, service, cluster, or another persistent label rather than relying on ephemeral hostnames where possible.

The best approach is to design the alias to include a stable identifier that matches your operational model (service, cluster, environment, etc.), rather than relying on the integration key itself.