PRTG integration (Python script)

Richard Norman March 24, 2021

Hi community,
I am trying to get acknowledgement of alerts in Opsgenie to trigger (via HTTP API) acknowledgement in PRTG of the initial alert.  
I have everything else in terms of the HTTP PRTG Opsgenie integration functional.

The issue I seem to be having revolves around the local OEC service on the PRTG server having issues running the python script.  The script in use is the one referred to from the Atlassian PRTG integration article and available from Github.

 

The error I am getting is pasted below and seems that certain parts of the ActionExecutor script are not understood.
I am not a python coder so there is every chance I have completely misunderstood what manipulation I need to do with the script.

Any insight would be great

Thanks

 

Rich

 

Python errors:

DEBUG[2021-03-24T13:42:13.199198Z] Job[a658b2e9-7884-41b8-a631-fbc3982971df] has been processed by worker[d15b95b2-b14e-448b-8a83-1e34eeb67723].
DEBUG[2021-03-24T13:42:13.3519763Z] Message[289159ad-e8f5-4456-88c6-3d158e31d534] is deleted from the queue[us-west-2].
DEBUG[2021-03-24T13:42:13.3880999Z] Successfully sent result of message[a658b2e9-7884-41b8-a631-fbc3982971df] to OpsGenie and it took 0.188902 seconds.
DEBUG[2021-03-24T13:42:13.4901267Z] Action[acknowledgeSensor] execution of message[289159ad-e8f5-4456-88c6-3d158e31d534] with entityId[5f8b3b61-70a1-45bc-af3f-b636af93e590-1616592719477] failed: exit status 1 Stderr: Traceback (most recent call last):
File "C:\opsgenie\scripts\actionExecutor.py", line 6, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

1 answer

0 votes
Connor Eyles
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 29, 2021

Hi Richard,

Would also like to inform you that I am not a python coder but can help with pointing you in the right direction in getting this working :)

The error message you are receiving is 'ModuleNotFoundError: No module named 'requests' 

To resolve this you will need to install the 'requests' module in your environment, check out this document here https://stackoverflow.com/questions/17309288/importerror-no-module-named-requests

This should help you with running the right command to install this module, let me know how you go!

Thanks,
Connor

Richard Norman April 13, 2021

Thanks Connor for your help on this one.  I have only just got around to having a good read through that article and installing the modules.  I am not in my comfort zone here but normally can apply enough savvy to get there in the end, so im still keen on getting it working.

Now I have applied the modules I dont seem to get any errors but when acknowledging an alert on Opsgenie the alert doesnt seem to get acknowledged on PRTG.  I have pasted below the output from the PRTG OEC integration service installed on the PRTG server.  I am grateful for your help so far.  If you have anything else you might be able to throw in to the ring I would be grateful as I want to see this through :-) Thanks

[37mDEBUG [0m[2021-04-13T15:15:58.5768097+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:16:13.5831573+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:16:28.574219+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:16:43.158568+01:00] Received 1 messages from the queue[us-west-2].
[37mDEBUG [0m[2021-04-13T15:16:43.1595668+01:00] Job[0b128283-ff33-4302-a104-5ffbb8c09ee2] is being submitted
[37mDEBUG [0m[2021-04-13T15:16:43.1605601+01:00] Job[0b128283-ff33-4302-a104-5ffbb8c09ee2] is submitted to worker[cc6ae6d4-8108-4311-bf25-cdd940047d92]
[37mDEBUG [0m[2021-04-13T15:16:43.5248695+01:00] Message[0b128283-ff33-4302-a104-5ffbb8c09ee2] is deleted from the queue[us-west-2].
[37mDEBUG [0m[2021-04-13T15:16:43.5748759+01:00] Current Worker: 4, Idle Worker: 3, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:16:44.4010725+01:00] Action[acknowledgeSensor] execution of message[0b128283-ff33-4302-a104-5ffbb8c09ee2] with entityId[80f53b00-ecf1-4f79-84a5-e77511b8a6d6-1618323335455] has been completed and it took 0.876208 seconds.
[37mDEBUG [0m[2021-04-13T15:16:44.4010725+01:00] Job[0b128283-ff33-4302-a104-5ffbb8c09ee2] has been processed by worker[cc6ae6d4-8108-4311-bf25-cdd940047d92].
[37mDEBUG [0m[2021-04-13T15:16:44.6028558+01:00] Successfully sent result of message[0b128283-ff33-4302-a104-5ffbb8c09ee2] to OpsGenie and it took 0.192868 seconds.
[37mDEBUG [0m[2021-04-13T15:16:58.5817922+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:17:13.5746428+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
[37mDEBUG [0m[2021-04-13T15:17:28.5740414+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0

Connor Eyles
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 13, 2021

Completely understand that! Well the log messages state "Action[acknowledgeSensor] execution of message[0b128283-ff33-4302-a104-5ffbb8c09ee2] with entityId[80f53b00-ecf1-4f79-84a5-e77511b8a6d6-1618323335455] has been completed"

Which tells me that it executed correctly! The next thing I would suggest is that there are no networking issues between your OEC server and your PRTG server and check the PRTG server for any logs to see if it has contacted PRTG correctly.

Another thing to check that the PRTG credentials are set correctly within the conf.json so that the script can correctly authenticate with PRTG.

Thanks,
Connor

Richard Norman April 20, 2021

Thanks Connor, 

I have done a quick check along these lines and I have actually installed the OEC service on the PRTG server so network connectivity shouldnt be an issue.  I don't see a firewall issue either as I assume if the PRTG http interface is available from the web then opsgenie should be able to speak to it OK.  Something I now think might be an issue is the account on PRTG opsgenie is using to try and acknowledge the alerts, and whether that account has the right access or permissions within PRTG to do so?  I have as such made the account a member of a group with full visibility of all devices and retested without luck.  Any further thoughts?

 

cheers

Rich

Connor Eyles
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 20, 2021

Hi @Richard Norman 

I think the next best action is that you raise a support ticket with us so we can gain access to your Opsgenie to review the logs from the Opsgenie point of view. To ensure that it is executing correctly!

https://support.atlassian.com/opsgenie/

Thanks,

Connor

Richard Norman April 21, 2021

Hey Connor.

I thought maybe having OEC run on the PRTG server might be causing issues accessing the PRTG web page (HTTPS offloaded via a load balancer)  To get around that I created an entry in the PRTG server local hosts file for the loopback to resolve to the PRTG web URL.  At this point I got a lot of errors as the OEC service didnt like the fact the PRTG server doesnt have a cert (that is why I offloaded the SSL as putting a cert on PRTG isnt straightforward.

Having proved that the OEC was now speaking internally to the apache server (to eliminate any networking issues or load balancer issues) I decided to go through the antiquated process of putting a cert on the server itself to get ris of the OEC SSL errors now seen.  I successfully managed to get rid of the errors afterward and success messages came back as before (pasted above) After all that, the back acknowledge to PRTG still isnt functional.  I have not raised a ticket with you guys before but I think now is the time. Can you please guide me through the process if you wouldnt mind?

 

Many thanks for all your help so far and moving forward.

 

Rich

Richard Norman April 21, 2021

FYI errors when cert didnt work

File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\ssl_.py", line 428, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\ssl_.py", line 472, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='prtgservice.surrey.ac.uk', port=443): Max retries exceeded with url: /index.htm/api/acknowledgealarm.htm?id=15856&ackmsg=Richard+Norman+acknowledged+alert%3A+%22%5BPRTG%5D+%5BPRTG+Network+Monitor+%28PRTGAPP01V%29%5D+AMONbalancer+Throughput+%28SNMP+Custom%29+Down%22+https%3A%2F%2Fopsg.in%2Fa%2Ft%2Funiversityofsurrey%2F9099&username=netops%40universityofsurrey.opsgenie.net&passhash=1408463140 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\opsgenie\scripts\actionExecutor.py", line 89, in <module>
main()
File "C:\opsgenie\scripts\actionExecutor.py", line 78, in main
response = requests.post(result_url, params=params, timeout=timeout)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\rn0017\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)


After cert installed on PRTG backend web server

DEBUG[2021-04-21T10:14:54.3774497+01:00] Current Worker: 4, Idle Worker: 4, Queue Size: 0, Queue load: 0
DEBUG[2021-04-21T10:15:08.6839897+01:00] Received 1 messages from the queue[us-west-2].
DEBUG[2021-04-21T10:15:08.6849936+01:00] Job[73e6790f-02e0-40e8-bcc0-9c0655557924] is being submitted
DEBUG[2021-04-21T10:15:08.6849936+01:00] Job[73e6790f-02e0-40e8-bcc0-9c0655557924] is submitted to worker[1794559e-8622-4395-8dc5-b02d18ba107c]
DEBUG[2021-04-21T10:15:09.0391197+01:00] Message[73e6790f-02e0-40e8-bcc0-9c0655557924] is deleted from the queue[us-west-2].
DEBUG[2021-04-21T10:15:09.3645657+01:00] Current Worker: 4, Idle Worker: 3, Queue Size: 0, Queue load: 0
DEBUG[2021-04-21T10:15:10.2086108+01:00] Action[acknowledgeSensor] execution of message[73e6790f-02e0-40e8-bcc0-9c0655557924] with entityId[17551ce8-e333-4998-88d6-3e102af38049-1618996472332] has been completed and it took 1.169483 seconds.
DEBUG[2021-04-21T10:15:10.2086108+01:00] Job[73e6790f-02e0-40e8-bcc0-9c0655557924] has been processed by worker[1794559e-8622-4395-8dc5-b02d18ba107c].
DEBUG[2021-04-21T10:15:10.3924782+01:00] Successfully sent result of message[73e6790f-02e0-40e8-bcc0-9c0655557924] to OpsGenie and it took 0.183866 seconds.

Connor Eyles
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 21, 2021

Hi @Richard Norman 

You can start by going to this URL https://support.atlassian.com/opsgenie/

Then click on the green "login to contact support" which will direct you to log into your Atlassian account.

Then start filling out the form for Opsgenie and support details!

I will keep my eye out on the support Queue so I can pick up the case smile

Thanks,

Connor

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events