OEC Scripting

[A.G. Rappe]

Are there any docs on how to create scripts for OEC and how to use them?  I have the OEC service running on a local server.  How do I use OEC?

Thanks,

AG

[connor mcguane · Accepted solution]

Yes and no.  There are some docs here, but they dont really have code examples.  You can google around and find some public Github repos with code examples.

As far as how to use the OEC, once installed you need a script for it to call.  That's basically all the OEC is, is a service that listens for Opsgenie alerts and then executes a script.

[A.G. Rappe]

I've heard and seen in videos where alerts can have an option to ping a down host or restart a service.  How do you create these and how do you have them show up in alerts?

[connor mcguane]

So you may be talking about Heartbeats, so I'll leave this here for you to explore that if that interests you.

Otherwise, I'll explain how our environment is setup to do restarts.  We have an OEC Service running on a host.  If an alert comes in called "Service XYZ not running..." that matches our OEC integration.  It then sends this alert's message to the OEC.  The OEC then runs a Powershell script.  That script takes the data, and calls to another service of ours that attempts to restart the service.  If we get a successful restart, that same powershell script will write a note to the calling alert saying it successfully restarted, and closes the alert.  If the restart failed, it will just write a note.  Then escalation will continue normally.  Calls triggering the OEC integration show up in the Activity Log like anything else.

Really, the OEC service itself doesnt do the heaving lifting, you must provide the script to give it a functionality.

[A.G. Rappe]

Thanks.  Sorry for being dense.  I understand about Heartbeats.  I just can't find any documentation on how to use OEC for anything other than AWS.  I want to build Powershell scripts that I can use with OEC, but I have no idea how to go about it.  I've seen in Youtube videos that you can build scripts so an alert can ping a downed server and include the results as an attached note, or restart a service with OEC.  Where do I go to learn how to configure this?

[connor mcguane]

Like I said, there are some examples around the internet of people's script in the OEC.  I admit, I was in your shoes about a year ago on this, so do worry (youre not dense).  It isnt very well explained how do anything useful with the OEC.  I hope the following helps.

So here is our pseudo-code example of the restart process I mentioned previously:

Make sure this Powershell script is in the scripts folder of the OEC, and all conf files point to the right places.

Firstly, we get the input of the triggering alert.

# Convert the JSON String sent to the OEC into a PowerShell Object
$input = $args[1]
$alertObj = $input | ConvertFrom-Json

We then get the pieces of information we need.

$assetKey = $alertObj.params.alertDetails.Asset
$alertID = $alertObj.params.alertID
$originalPriority = $alertObj.params.alertDetails.OriginalPriority

 We then do two things in this example.  We update the triggering alert with a note.  And then use the information we parsed to send to another API.

#default API key
$headers = @{
"Authorization" = "GenieKey 14********************bbd2c1"
}
$oGNotesURI = "https://api.opsgenie.com/v2/alerts/$alertId/notes"
$jsonBody = @{
note="this script triggered the OEC"
} | ConvertTo-Json
$ogResponse = Invoke-WebRequest -Uri $oGNotesURI -Method Post -Body $jsonBody -ContentType "application/json" -headers $headers -Proxy "http://serverproxy.*****.com:80

$externalURI = "https://******.com/api"
$body = @{
alertID=$alertID
priority=$originalPriority
key=$assetKey
}
$jsonBody = $body | ConvertTo-Json
$ogResponse = Invoke-WebRequest -Uri $externalURI -Method Post -Body $jsonBody -ContentType "application/json" -headers $headers -Proxy "http://serverproxy.*****.com:80"

 This is obviously not code complete, but hopefully gives an idea of how to work with an alert in Powershell.

[A.G. Rappe]

This is great!  At least I have a place to start.  Thanks!

[Nicola D_]

Hello. I think that OEC should be used fin combination with a  monitoring tool.

You can find a lot of OEC integration with common monitoring platform (nagios, Icinga2, zabbix etc) 

Here you can find a detailed use case https://www.neteye-blog.com/2020/10/incident-response-in-neteye-with-atlassian-opsgenie/

Cheers 

[A.G. Rappe]

Thanks for the input.  We have an on-site system for monitoring.  We've plugged in Opsgenie to sort out the 'noise' alert emails.  I have a working integration with this system.  Since the agent on the endpoint can sometimes crash, I'm trying to build an action that allows the on-call engineer to ping the down system and/or restart the agent service from the alert.  I've seen in sales videos that this is possible, but the actual contents of the action script are elusive to say the least.  I've seen scripts that add notes, close, create, acknowledge, etc. but none that accomplish a ping or restart-service command.

[Tom Russell]

Your script can do a ping in multiple ways using multiple languages. I could show you the code our script uses to restart a service, but all it does is call a third-party tool API. Unless you have the exact same set of tools and configurations, it's useless to you.

I would recommend you find examples out there (independent of Opsgenie) that help you to create a script that accepts the parameters you need, does the ping or restart, and returns some type text status. If you can get that far, the Opsgenie community can help you integrate that into your OEC environment.

[Nicola D_]

For the OPSGenie Alarm integration if you do not have a monitoring tool you could evaluate LAMP CLI of OPSGenie (https://docs.opsgenie.com/docs/lamp-command-line-interface-for-opsgenie)

Regarding the service command it depends on Technology and Architecture of the environment you need to work with.