How to solve Opsgenie issues using the debug logs

[#1]

The Opsgenie debug logs can be extremely useful when Opsgenie isn't working the way you expect.

The debug logs can tell/show you:

• Why an alert wasn't created
• Why an alert wasn't closed or acknowledged
• Why an alert has missing or incorrect information
• Why an outgoing action wasn't sent to an external service
• Configuration changes, who performed them, and when
• Raw, unprocessed incoming payloads from your integrations
• Processed, and parsed payloads showing how your data translates to Opsgenie-required fields
• Much more

A few things to note about the debug logs:

• The logs show up to the last 13 days
• There is a custom time filter that is useful if you know when an event was sent to Opsgenie
• Time stamps on logs match your profile settings - i.e. if your profile is set to EST, the logs will be in EST time. 
• JSON formatted data can be expanded by clicking on the arrows
• Search operators including ‘AND’ ‘OR’ and ‘NOT’ are available
• Wildcards including ‘*’ and ‘?’ can be used to match multiple or a single character, respectively.

One common occurrence that customers experience is when no alert is created after an integration sends a payload to Opsgenie. If the payload reaches Opsgenie, it will always be logged. So, we can search the logs to see why the alert wasn’t created. 

If you have specific values to search the logs with, that can make things easier - for example, if you know the alias that was sent to Opsgenie, you can put it in quotes and search the logs and it should turn up the incoming payload. 

Searching the logs by Integration ID

If you don’t have any of the data that was sent to Opsgenie available to you, one option is to use the integration ID. You can get this by going to the integration page and copying the ID at the end of the URL. Again, you will want to put this in quotes when you search for it. 

Using the custom time filter

If you have many logs to look through but you know when a payload was sent to Opsgenie, you can use the ‘custom time range’ filter. After you enter a time and search, the time you specified will be at the beginning of the logs, so you will need to click the double-right arrow at the bottom of the page to get to the logs for that time.

Error messages

Once you find the payload, you should see a clear message stating why an alert was not created. One common reason is that the payload did not match any of the integration filters, in which case the error message will state that “no matching actions” were found. Many other error messages will also state very clearly the cause of the issue. 

Alerts not being closed or acknowledged 

Incoming payloads for ‘close’ and ‘acknowledge’ actions can also be discarded if they do not match any of the integration’s configured ‘close’ or ‘acknowledge’ actions or if they do not match the alias of the alert they are intended to close. You can read more about this topic here. 

Troubleshooting missing or incorrect alert data using the unprocessed payload

If you have a situation where an alert was created but the fields do not appear to be parsed correctly (either a missing or incorrect value), you can view the raw, incoming payload to see exactly what was sent to Opsgenie, which will help you determine why the value is incorrect. 

For each incoming alert, there are several logs created. One will show the raw payload and another will show the parsed payload. To find the raw payload we can search with the alert’s ‘incomingDataId’ in quotes - which can be found at the very beginning of the alert’s activity log. 

The debug logs are the most powerful troubleshooting tool in Opsgenie and are heavily used by the support team to solve customer issues. Getting comfortable with this tool will help you quickly solve problems in Opsgenie. 

Here is a list of useful debug log queries:

https://community.atlassian.com/t5/Opsgenie-articles/Useful-debug-log-queries/ba-p/2444507#M372

In order to ensure that we continue to provide useful content, please let us know if this Article is helpful (Thumbs Up/Down). Also, to help us improve, feel free to provide additional feedback (directly in the community).

[Emil Varona]

Is there a way that the raw JSON I see in the log entries can be retrieved using the API? The UI is extremely cumbersome for retrieving data from Opsgenie and I use the raw JSON to troubleshoot our integrations. TIA.