PRTG Integration suddenly stopped working two nights ago.

[Martina Bracek]

Getting sporadic responses from support, and am looking for help from the community because we are now reaching critical service interruption. 

We have complex alerting and scheduling set up in OpsGenie which rely on our PRTG API integrations. Over 40hrs ago, in the middle of the night, calls from our PRTG server stopped generating alerts. We also could find no trace of a failed API call from our OpsGenie logs.

During troubleshooting, we noted the odd integration requests would be received, but now with completely blank header parameters.

When inspecting the API call from our PRTG server, I see the following error: HTTP/1.1 426 Upgrade Required (Sensor/Source/ID: 2321/3483/1)

Nothing has changed on our systems. We refreshed the API, checked our configuration settings, recreated the integration on both sides, confirmed the service is using tls 1.2. Patched and rebooted our server.

[Allen Barnard · Accepted solution]

Hi guys,

Just to follow up on the Powershell script that @Martina Bracek mentioned. This script can be used by customers using PRTG on Windows Server 2012 + (supporting PowerShell 3 with executionpolicy remote signed enabled).  The script is yet to be merged with the official Opsgenie-integrations repo on git hub, for now, you can get it from my unofficial branch on github https://github.com/AllenCraigBarnard/opsgenie-integration/tree/master/prtg/Powershell_Integration 

Please note this is not intended for permanent use this should be regarded as a temporary workaround while we seek a permanent solution.

What are the dependencies and what OS is supported?

• Windows Server with Powershell 3 or above. Note PRTG Hosted is not supported.

• Since it is not possible to set Powershell’s -executionpolicy bypass parameter, the user needs to ensure that the execution policy is set to remotesigned 

• This script has only been tested with the latest version of PRTG, previous versions might not be supported.

How to install the script?

1. Download the Opsgenie.ps1 and Powershell_parameters files from this page.

2. Copy to this location “C:\Program Files (x86)\PRTG Network Monitor\Notifications\EXE”

3. Open PRTG, go to “Setup” >> “Account Settings“ >> “Notification Templates”

4. Click the blue plus sign located to the right of the “Notification Templates” table, then click “Add Notification Template”

5. Give the template a name “OpsgenieScript“ and make sure its status is “Started“

   

6. Toggle on the option “Execute Program“, from the “Program file“ drop-down list select “Opsgenie.ps1“. Open the file Powershell_parameters that you downloaded from this page, copy all its content and paste it into the “Parameters“ field, replace [API URL] with the API URL contained in your PRTG integration settings in the Opsgenie UI.
   
   In “Domain or Computer name“ you will need to specify Windows Domain name (if the PRTG server is an Active Directory is domain member) or Computer name (if a non Active Directory computer)
   
   Next, specify a username and password credentials for running the script, please do not use an Administrator user, please create a service account user if possible.

7. Lastly, we need to configure triggers. In PRTG select “Devices“ then click “Notification Triggers“, Click the blue plus sign located to the right of the “Notification Triggers” table, then click “Add State trigger” configure the following settings:

When sensor state is Down for at least 60 seconds, perform  OpsgenieScript

When sensor state is Down for at least 300 seconds, perform  OpsgenieScript and repeat every 5 minutes

When sensor leaves Down state after a notification was triggered, perform  OpsgenieScript

Create a second state trigger that handles warnings.

When sensor state is Warning for at least 60 seconds, perform  OpsgenieScript

When sensor state is Warning  for at least 300 seconds, perform  OpsgenieScript and repeat every 5 minutes

When sensor leaves Warning state after a notification was triggered, perform  OpsgenieScript

[Martina Bracek · Accepted solution]

Hey guys. OpsGenie support did end up getting back to us with a workaround. They provided a PS script to send the WebRequest, and we updated our notification template in PRTG to call the script on trigger. 

[Rzuniga]

Hi martina, How could I contact you to help me with a couple of questions about the operation of opsenginie?

[Andrew Crerand]

I’m having the same issue. It suddenly started working again for me this morning, but given the responses from support on both sides, it doesn’t inspire confidence it’ll keep working. Atlassian says it’s due to them disabling TLS 1.1, but PRTG says it’s because OpsGenie now requires HTTP 2.0 and that is not supported by PRTG yet. Can I ask how you confirmed your PRTG server is using TLS 1.2? 

[Martina Bracek]

Under system settings, the server shows it is using TLS 1.2. We also used Wireshark to observe outgoing webhooks and they were using TLS 1.2.

[Andrew Crerand]

Thanks Martina! 

[Roy Meister]

Hi, we see the exaxt same issue in our PRTG instance.

The last successfull call to the API I can find was 24.11.2020 16:07:16 (CET). So somewhere after this something changed.

Seems like a HTTP/2 vs HTTP/1.1 issue.