Running a python script on a confluence page

[Bernd Steffens]

Hi all,

I have a python script that collects data from different sources and creates a pdf file afterwards, like an evaluation. Is it possible to do this on a created confluence page? My idea was to create a macro, this macro has a button and on click this button starts the python script and the page shows the result pdf. I hope it is clear what I am trying to achieve?

All I found out so far is, that it is not possible to directly run a python script on a confluence page and I need some kind of "bridge" to do so. Unfortunately I didn't find any further information or tutorials on how to do that exactly... So my question is, if someone here in the community has done something like that before and can push me in the right direction where to start or what alternatives are possible?

Thanks in advance!

[DPKJ · Accepted solution]

@Bernd SteffensRunning python directly from SDK plugin is possible but it is not recommended as it voids all the security provided by JVM in Confluence.

But still here are some things you can do, for running python code on button click via confluence,

• Create a macro that generates 'Button' and some Javascript to go with the Button as output, you can also add button/link at various other places like page menu, Check these SDK documentation for this,
  • For making macro - https://developer.atlassian.com/server/confluence/macro-module/
  • For serving JS/CSS - https://developer.atlassian.com/server/confluence/web-resource-module/
  • For adding button/link to page - https://developer.atlassian.com/server/confluence/web-ui-modules/
• Next on click of that button, you need to call some back-end API, for this you will need to create a custom REST endpoint, (you can pass page details to this API)
  • https://developer.atlassian.com/server/confluence/rest-module/
• In this REST API, you can call external python program using,
  • Java ProcessBuilder - https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
  • Java Runtime Process - https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
  • You can find more options like this.
• This python program must return you location of pdf file, which you can attach to page, or provide link to use for download.

But it will really nice,

• if you can write replicated your python program in Java, and
• You can also explore using Jython - https://www.jython.org/

Hopefully this resolves all your doubts.

[Sven Schatter _Lively Apps_ · Accepted solution]

Hi @Bernd Steffens,

I have a python script that collects data from different sources and creates a pdf file afterwards, like an evaluation. Is it possible to do this on a created confluence page? My idea was to create a macro, this macro has a button and on click this button starts the python script and the page shows the result pdf. I hope it is clear what I am trying to achieve?

Generally, this should be possible. There's a million ways to achieve this, each with their own benefits and disadvantages, so the hardest thing here is probably to find the way that works best in your situation.

All I found out so far is, that it is not possible to directly run a python script on a confluence page

Kind of correct. You could probably build your own Confluence P2 Plugin and use something like the Graal Polyglot API to execute your Python code on the Confluence Backend. However, this is probably not a very good idea for many different reasons. (Performance, Experimental Feature, Security, etc.) And if you're already implementing a P2 Java Plugin anyway to be able to run Python code on your Confluence Backend... you might as well just rewrite your Python code in Java, as that would probably be much less of a hassle.

I need some kind of "bridge" to do so. Unfortunately I didn't find any further information or tutorials on how to do that exactly... So my question is, if someone here in the community has done something like that before and can push me in the right direction where to start or what alternatives are possible?

The easiest of all "bridges" in your case would probably be a REST service. I would deploy your Python script somewhere in the form of a web server, so it can be invoked via a HTTP/REST call. In Confluence you could then write a macro via a P2 plugin. The macro would simply display a button, as you said, and on click, call a REST endpoint of your P2 plugin.

Since the best way to display a PDF in Confluence is probably to simply attach it to a page, the REST endpoint would fetch the PDF from your Python Script Server and attach it to the page that the button was clicked on. Finally, you could show the user the attached PDF, for example by opening the link to it in a new tab. (Of course you could also display it directly in the page in place of the button, but that's probably not quite worth the effort that would go into something like that) I've quickly drawn a pseudo diagram that hopefully makes more sense of what I'm saying.

If this sounds super complicated and you have no idea where to start, you should probably try to find a local Atlassian Solution Partner that can do this for you. Implementing stuff like this is pretty much the everyday job of a Solution Partner that does customizations. :)

Best regards,
Sven

[Sven Schatter _Lively Apps_]

The answer by @DPKJ is pretty much the same solution as what I have suggested, except that in this case the Python code is executed on the same machine as Confluence, whereas in my example the Python code is executed on another machine. Both solutions have advantages and disadvantages, so here's a short list to help you decide what you want to do (as neither of both is just better).

Running the Python code on the same machine as Confluence and invoking it via an external process:

• Requires you to have Python installed on that machine. (in most cases probably already a given)
• Uses the same resources (CPU / RAM / Disk Space) as the Confluence server which could be a bad thing depending on the size of your Confluence instance.
• Updating the Python script would require you to switch it out on the Confluence Server.
• If you're passing some kind of parameters to the Python script and are not sanitizing them thoroughly this could quite easily become a security risk.
• Probably a bit easier to implement.

I would probably do it this way if you're doing this for a smaller organization (like 100 people or less) where a hit on Confluence performance might not be as bad and "slight security risks" are not really a problem.

Running the Python code on another machine:

• Requires you to maintain and pay for an extra machine. Also probably requires some network setup / configuration.
• No changes to the Confluence Server (except installing a plugin) and no sharing of resources.
• Updating the Python Script would require you to redeploy your Python Server. (Would be very smooth with a CI/CD pipeline)
• Probably less of a security risk, since the Python code is executed on another machine. And the Python machine could also be locked down network wise, have its own authentication mechansim, etc.
• Probably a bit harder to implement.

I would probably do it this way if you're doing this for an enterprise.

Best regards,
Sven

[Bernd Steffens]

Thank you for this advice. I spoke with our admin and the final implementation will be done like you described in your second option, because of your mentioned security reasons. But for the start and understanding part, accomplishment on your first version is the goal for now. Especially regarding easier implementation which is better for learning and teaching.

[Bernd Steffens]

Thank you for your answers @both of you. I read your recommendations and links you provided. At first it is a lot of information but it made me think. I try to accomplish this task as personal learning. Later on I want to "teach" my co-workers on how to achieve the desired goal. Because we will have other cases with python scripts and including them in confluence later on. So getting someone from the "outside" or using another programming language unfortunately is not an option.

That's why, my way will be creating a macro with a button which "activates" the python script. So far I have my macro and a simple button on the page with no function. Now I'm trying to figure out the "calling some back-end API" part. Is it possible for you to give further details on this? I find the atlassian documentation on the REST module and API not very helpful in my case. Or maybe there is a more understandable tutorial for connecting confluence macro via REST API with python out there, which I didn't found so far?

Thank you very much.

[Sven Schatter _Lively Apps_]

If you can't get much out of the documentation it might be best for you to just look at a working example. There are Marketplace Vendors that open source their apps, such as TNG with their Multivote for Confluence app. Since a vote macro is more or less just a button that makes a REST request this might be an excellent example for you to look at.

Here you can look at what they're doing in JS and here you can look at the REST implementation in the backend.

[Bernd Steffens]

Thank you for that hint. Seems like a small macro, but the code behind it is not so small. ;)

Maybe I have to mention that I am relatively new to Java/Javascript/Confluence macro development. I understand that in the .js file the "look" of the macro is defined, e.g. how the voting table is constructed. I don't understand everything, but most of it. For me the more interesting part probably is in the .java file. And here it is not so clear for me anymore. Actually I don't really see the connection to the REST API. What I see there looks to me like some kind of error handling, e.g. wrong inputs, site manipulation, etc. Or am I wrong?

[Sven Schatter _Lively Apps_]

The really important parts are the annotations of the method:

@POST
@Produces({MediaType.APPLICATION_JSON}) @Path("/page/{pageId}/table/{tableId}/item/{itemId}")

This tells you that the method will accept HTTP POST requests, that it will be accessible via the URL specified in the Path annotation and that it will return JSON to the caller. What happens inside of the method is purely up to you (and yes, input sanitizing and error handling is usually a big part of this).

In JS you can see that they're then calling this URL with an Ajax request:

$.ajax({ type:"POST", dataType:"json", url: url, ... })

And the response of the REST call is then handled in the success handler that is passed to the ajax function.

If this is super confusing for you, you might wanna read up on some general HTTP & REST basics first. Looks like this article could be a good starting point.

[Bernd Steffens]

Thank you very much, that was very helpful for basic understanding. I think I understood the concept of REST and how to handle it now. Theoretically this is what I have to do:

• macro on confluence page -> button on click sends a GET request to a REST endpoint
• from there I can make an external call to my python file which executes and generates a pdf file
• that file is then at the REST endpoint and because of my GET request accessible in my confluence page

Is that alright so far?

There are just two points I can't really wrap my mind around .... First I want to test the whole thing on a local confluence server. Is it then possible to access the python file on a local machine? Can it then be placed in the macro directory?

And how or where do I create my REST endpoint?

As I said, I understand the basic functionality, but having problems now with the right implementation.

If you could enlighten me a little bit more, that would be very helpful. Thanks in advance!

I just had a thought... maybe for better understanding on my side it is better to don't use a python file, but rather just "call" a test.txt in my local macro folder and include it on my confluence page. Or would this not be helpful for my actual task?

[DPKJ]

@Bernd Steffens  You are correct about the approach.

For storage of script I would suggest you to put it in 'resources' directory of your plugin. This way you can validate it during plugin development, and can be easily upgraded/installed with plugin on any Confluence server you want.

Two more thing,

1. If you python program is long running and it will take time to generate PDF,
   1. you should not hold API response,
   2. try to run it in JOB/Queue model
   3. Poll from front-end for PDF
2. If you are struggling too much and can't ask for help from Atlassian Partners, I can give you a sample implementation (in couple of days, once I get some time), but make sure you test it before deploying.
   1. For this you need to tell me where you want to store generate PDF (can this be attachment to page from which macro was called?)

[Bernd Steffens]

Thank you for your reply.

To 1. The final python Script is not that big. It is planed that it collects text data from 10-20 internal HTML sites. So it shouldn't be too time consuming. But if we notice perfomance issues later on, I will re-check with your mentioned options.

To 2. If that is possible for you, it would be very nice. Learning from a working example is very helpful in my opinion! And yes, it is planed to attach the PDF directly to the confluence page from where it is called.

Thanks again!

[DPKJ]

@Bernd Steffens  I will provide you with some sort of sample add-on in a day or two.

[DPKJ]

@Bernd Steffens  Here you go - https://bitbucket.org/jangidd/external-script-executor-macro-for-confluence

Your developers can look at the code and modify it for production and better use.

Hopefully this helps you.

Macro selection screen,

This will visible on view screen,

[Bernd Steffens]

Wow, thank you very much. So far I was only able to take a quick look at the macro and the code. I have to take a closer look later on. Probably will have some questions ... But for now thank you very much!

[DPKJ]

@Bernd Steffens  Enjoy coding!