How to divert rest api calls from external applications to a particular node

[Jonnada Kiran]

We have a 4 node cluster for JIRA. We are using F5 load balancer to distribute traffic. Due to increase in rest api calls from external applications we are experiencing some performance issues. So, we want divert rest api calls from external applications to Node 4. I read this https://confluence.atlassian.com/enterprise/traffic-distribution-with-atlassian-data-center-895912660.html. But, not sure how to implement in our F5 load balancer. Can someone please give some detailed steps to implement this? Thank You.

[Sanu Soman]

Hi Jonnada,

Did you made it worked? I am also trying to setup same with F5 load balncer.

Please share the details if you done so?

Thanks,

Sanu P Soman

[Jonnada Kiran]

@Sanu Soman Hi. I have implemented it. It works great. Implement the below irule in F5. Thanks.

when HTTP_REQUEST {

log local0. "New code"
set referrer_host [ URI::host [HTTP::header value Referer]]
log local0. "host: [HTTP::host]"
log local0. "uri: [HTTP::uri]"
log local0. "path: [HTTP::path]"
log local0. "Referer: $referrer_host"


if { ([HTTP::header exists "Referer"]) } {

if { $referrer_host eq "" } {
 set referrer_host "none"
 }

if { !($referrer_host contains "jiratest.corp.chartercom.com") } {

log local0. "referrer is not equal to jira"
 if { ([HTTP::uri] contains "/rest/") } {
log local0. "referrer is not equal to jira and contain rest"
 pool JIRA-test-external-pool
 } else {
 pool JIRA-test-pool
 }
 
 } else {
 log local0. "referer is jira"
 pool JIRA-test-pool
 }

} else {
log local0. "Referer does not exist"

if { !([HTTP::uri] contains "/rest/") } {
log local0. "referrer is not equal to jira and contain rest"
 pool JIRA-test-pool
 } else {
 pool JIRA-test-external-pool
 }
 }


}

[Sanu Soman]

Great and many thanks for sharing it.

Is your JIRA running with SSL enabled? If so, do we need to terminate SSL at F5 level to configure this?

[Jonnada Kiran]

I dont know about SSL.

[Thomas Deiler]

Dear @Jonnada Kiran,

you have to terminate SSL on F5 otherwise you cannot filter, because the path information will be encrypted.

So long

Thomas

[Sachin]

Hi Thomas,

We have the same requirement where we want to redirect external Rest API only to specific node we have 4 node and we want to redirect external REST traffic to node 4.

We have written iRule for that but we observed few thing

1. Jira becomes slow.

2. Xray plugin is not working as expected like "Test" tab is not appear on the screen.

[Sachin]

Thomas,

what do you mean "terminate the SSL on F5"? because we have SSL implemented on F5.

[Thomas Deiler]

Dear @Sachin,

as I wrote, if you will try to load balance HTTP traffic depending on the path information of the URL, it may not be encrypted.

This slow Jira will result (I guess) from Jira's own communication over the REST API. Best do not balance this to only one node.

For the XRay Plugin, best ask the vendor. They should know about their plugin engine.

So long

Thomas

[Orit Nachshon]

Hi Thomas,

Thanks for reply.

So what you r saying is we should not redirect REST API call to specific one node if we have SSL on F5?

And if we want to redirect then we have to skip ssl?

Thanks,

Sachin

[Jonnada Kiran]

@Thomas Deiler@Sachin@Sanu Soman Sorry for replying late. Redirecting traffic to a node raised indexing problems between nodes. 

[Orit Nachshon]

@Jonnada Kiran:  Did you experience this? Did Atlassian confirm about this? what is solution for this then?

[Jonnada Kiran]

Arlassian recommended not to divert rest calls till they find a solution 

[Orit Nachshon]

@Jonnada Kiran Thanks for reply. Does they have created bug or any issue for their developer for this?

[Andriy Yakovlev _Atlassian_]

Hi all, 

I see that this is some confusion around LoadBalancing, so to prevent the uncertainty please let me clarify some points. :)  

• To do session affinity at Loadbalancer and redirect traffic based on URL path (/REST), you would need to see request plain text, so as Thomas mentioned you need to terminate SSL on F5 (do SSL offloading). This is just a technical requirement. 
• This is very crucial to make sure that you redirect traffic only from External application (bot), not from the normal user. In another case, you will be break session affinity which leads to different functional and performance problems. 
•  Regarding Redirecting traffic to a node raised indexing problems between nodes. It's hard to comment without context, but in general, redirecting some traffic to the separate node does differ from any other normal setup. We have many clients running this without any problems.
  • Speculating here: if dedicated node would receive too many changes, those changes would need to be propagated to other nodes (load is skewed) and in case cluster doesn't have enough capacity that would case indexing delays. That being said, this is not specific to Redirecting traffic to a node but like a genetic balancing load problem. 

How you can check that session affinity is fine. 

• Check reply headers from Jira and find x-anodeid, eg:

x-anodeid: i-02b2b80fafcf9ad1d

• make sure it's the same for all requests 

Hope this helps

Cheers

[Orit Nachshon]

@Andriy Yakovlev _Atlassian_ 

We have written the same iRule which target only external REST API.

Can you please explain 2nd point i.e  "This is very crucial to make sure that you redirect traffic only from External application (bot), not from the normal user. In another case, you will be break session affinity which leads to different functional and performance problems."

In that you mentioned *Not from the normal user* what does it mean? and how do we check this?

Thanks,

Sachin

[Andriy Yakovlev _Atlassian_]

Hi  @Orit Nachshon

Sure, please let me help with this part:

>In that you mentioned *Not from the normal user* what does it mean?

That means, that user sessions should not be part of the redirect config. They should always talk to the same node.

>and how do we check this?

see the section (How you can check that session affinity is fine. ) in the comment above 

Cheers. 

[Orit Nachshon]

@Andriy Yakovlev _Atlassian_@Thomas Deiler

This is the iRule we have implemented. Can you please check. it?

when
HTTP_REQUEST {
 if

{ !([HTTP::header value Referer] contains "jirapbguat") and ([HTTP::uri] contains "/rest") and ([HTTP::uri] ne "/login.jsp") }

{   node 10.233.128.120 8080  }

}

Thanks,
Sachin

[Thomas Deiler]

Dear @Jonnada Kiran,

separating REST from 'normal' calls can be done by analyzing the header information of HTTP requests, sent to the load balancer from the network. Everything that carries /rest/api/ in the destination URL is directed to a specific node.

As your company has invested in premium product, there should be a comprehensive documentation about F5 traffic balancing. I cannot help further, I am not familiar with this product.

So long

Thomas

[Thomas Deiler]

@Jonnada Kiran and @Sanu Soman,

keep in mind, that Jira is communicating with itself via REST. So best you do not balance requests from Jira.

Jira will request the F5 from its external network interface and not communicate from localhost to localhost.

This is not valid for Confluence, only Jira.

So long

Thomas 

[Andriy Yakovlev _Atlassian_]

Hi all, 

To clarify @Thomas Deiler answer:

Everything that carries /rest/api/ in the destination URL is directed to a specific node.

Jira UI operations also use same REST API, so you should be careful here and redirect traffic to specific Node4 only from External application (bot), not from normal user. In another case you will have user session expired all the time. 

There is no single way how you can do this, a couple of options to name:

•  based on Referer, as suggested in traffic-distribution-with-atlassian-data-center

• If referrer != app.yourdomain.com and the request ~ /rest:
   If the page requested == /login.jsp:
      Direct traffic to Node 1 or Node 2 or Node 3 //normal traffic
   Else:
      Direct to Node 4 //external REST API traffic

• Based on SrcIP
• Based on UserAgent

Hope this helps.