Having issues with installation

[Balazs Tarnok]

Hey All!

First of all, thank you for working on this great, new deployment option! In my organization we are also trying to go through the installation process but faced a few issues. It would be also really nice if we could have a call sometime, if it is possible.

We are using EKS on AWS, I just have a bit of experience with these technologies so maybe going to have a few basic questions too.

We tried to use different nginx installations: the one with helm in your documentation (classic lb) and two others from the original site (alb,nlb) (https://kubernetes.github.io/ingress-nginx/deploy/#aws). We realized that all these installations are creating internet-facing load balancers which is forbidden in our organization. My knowledge is basic about networking so I would like to know that would it cause any problem or further changes in other parts of your recommended installation if we would use an internal load balancer? 

I also tried to install into my personal aws account where internet-facing load balancers are allowed. I installed a classic load balancer with the recommanded helm commands from here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/examples/ingress/INGRESS_NGINX.md. Then continued with the installation guide from here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/INSTALLATION.md. Skipped step 3,4,5 just to have the most basic installation. A pod, service and statefulset was created but the pod stucks in pending status. Is step 3,4,5 required to have a running pod?

Then I tried to deploy the ingress (step 4) too, changed values.yaml but the pod is still in pending status, the nginx controller 'could not find any active endpoint'. Is it a must have to set the 'use-forwarded-headers=true' in the configmap before the nginx controller installation? As it states here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/CONFIGURATION.md

As I mentioned at the beginning, I have many questions, maybe even really basic ones but thank you for your help in advance, it would be great to have a discussion with you!

Best regards,

Balazs

[Dylan Rathbone · Accepted solution]

Hi Balazs,

Thanks for showing an interest in the DC Helm charts! We can certainly provide some pointers that might assist in getting you moving along. First thing I recommend is focussing on the pod and its pending state, steps 3,4,5 are definitely not needed for kicking the tyres and evaluating a DC product provisioned via the charts.

By the way what product are you trying to install?

Back to the pending pod, I'd recommend using kubectl describe to get some lower level details as to why your pod is pending i.e.

kubectl describe pod <pod-name> -n <namespace>

Hopefully the events listing will shed some light on the problem. You can also use this official guide for some handy tips on debugging deployments. 

Let's get these details first and work from there 

As for provisioning an internal load balancer with ingress-nginx this can be done by supplying custom config when deploying. I'd recommend having a read through the offical readme for some more detail. What you're after I believe is something like this:

controller:
  service:
    internal:
      enabled: true
    annotations:
      "service.beta.kubernetes.io/aws-load-balancer-internal": "true"

and then use this config as follows:

helm install ingress ingress-nginx/ingress-nginx --values custom-config.yaml

[Balazs Tarnok]

Thank you very much for your quick reply Dylan!

I could figure out that the pods were pending because there were not suffiecient memory and cpu so had to change the instance type of the nodes. In my personal account for developing I am trying to install confluence. But in our organization account we would like to deploy confluence, jira and crowd. We have there two m5.4xlarge type nodes. Do you have any recommended node type for our use case with these 3 applications?

• In my account now I have a running pod with some log4j errors:

log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".

• I created the ingress, there are also endpoints:

confluence-datacenter 192.168.26.123:8090,192.168.26.123:5701

• These are the event logs

37m Normal Scheduled pod/confluence-datacenter-0 Successfully assigned datacenter/confluence-datacenter-0 to ip-192-168-30-206.eu-west-1.compute.internal
37m Normal Pulling pod/confluence-datacenter-0 Pulling image "atlassian/confluence-server:7.12.3-jdk11"
36m Normal Pulled pod/confluence-datacenter-0 Successfully pulled image "atlassian/confluence-server:7.12.3-jdk11" in 10.906763879s
36m Normal Created pod/confluence-datacenter-0 Created container confluence
36m Normal Started pod/confluence-datacenter-0 Started container confluence
36m Warning Unhealthy pod/confluence-datacenter-0 Readiness probe failed: Get "http://192.168.26.123:8090/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
36m Normal Sync ingress/confluence-datacenter-setup Scheduled for sync
36m Normal Sync ingress/confluence-datacenter Scheduled for sync
37m Normal SuccessfulCreate statefulset/confluence-datacenter create Pod confluence-datacenter-0 in StatefulSet confluence-datacenter successfu

I created a route53 record that would route traffic to the classic load balancer: dualstack.a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com. The record name is: confluence.datacenter.proba.com so in the values. yaml I set the ingress.host to this value.

When I try to load this page I get: ERR_NAME_NOT_RESOLVED

• Here is the nginx controller pod log:

W0727 09:46:40.206676 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:40.245260 6 main.go:112] "successfully validated configuration, accepting" ingress="confluence-datacenter/datacenter"
I0727 09:46:40.270813 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter", UID:"d080fa51-02e6-4a08-afe5-72e0d33e0de9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7120", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0727 09:46:40.315199 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:40.362342 6 main.go:112] "successfully validated configuration, accepting" ingress="confluence-datacenter-setup/datacenter"
I0727 09:46:40.380022 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter-setup", UID:"9fffb04c-a000-4bcb-9382-29a9e38301fb", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7125", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0727 09:46:43.519960 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:43.523220 6 controller.go:146] "Configuration changes detected, backend reload required"
I0727 09:46:43.610188 6 controller.go:163] "Backend successfully reloaded"
I0727 09:46:43.610434 6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-nginx-controller-d9458694b-rtzgf", UID:"5808b0ba-331a-4d18-91ad-3de3d86704bd", APIVersion:"v1", ResourceVersion:"3392", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0727 09:46:46.853322 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
W0727 09:47:01.934715 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:47:40.159965 6 status.go:284] "updating Ingress status" namespace="datacenter" ingress="confluence-datacenter" currentValue=[] newValue=[{IP: Hostname:a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com Ports:[]}]
I0727 09:47:40.159965 6 status.go:284] "updating Ingress status" namespace="datacenter" ingress="confluence-datacenter-setup" currentValue=[] newValue=[{IP: Hostname:a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com Ports:[]}]
I0727 09:47:40.185936 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter-setup", UID:"9fffb04c-a000-4bcb-9382-29a9e38301fb", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7358", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0727 09:47:40.186885 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter", UID:"d080fa51-02e6-4a08-afe5-72e0d33e0de9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7359", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

[Balazs Tarnok]

I could get a bit further:
If I use the external ip of the load balancer in values.yaml at ingress.host then I can load the confluence setup page in a browser with the external ip. The browser still showing that it is not secure, I am a bit confused with the cert-manager and tls part.

Based on this I am having a problem with my route53 record and dns registration.

[Dylan Rathbone]

Hey Balazs,

Glad to hear you got your pods provisioned, I had a suspicion that K8s might have been having scheduling issues due to insufficient resources. Getting you're Pods scheduled is a good start, you're on your way now! . Just as a side note, you can tune the requested cpu and memory amounts required by each Pod. Doing this may have allowed you to provision Confluence to your existing nodes after all, see here for details.

With regards recommendations for the type of EC2 instance to use with Confluence, please take a look at this guide it should help give you an idea of what to use based on data sizes.

Balazs, it sounds like you are now using an external load balancer? If you want to externalize the Confluence service from the cluster then an external load balancer will be needed. Your DNS issues could be related to a number of things, namely pending verification by AWS of the domain name itself i.e. ".proba.com". If this has not yet been verified then it could explain why you can hit the service on the external IP of the LB but not the record "confluence.datacenter.proba.com". Take a look at the dig command for verifying the domain name.

For certificate issuance and management, when used in conjunction with the Ingress example we supply, we have had success with cert-manager and Let'sEncrypt. This setup must be in place before installing Confluence with Helm. What happens is on Confluence install a once off pod is created by the cert-issuer which then requests a TLS cert from Let'sEncrypt to associate with the Confluence service via the Ingress. This is only an example of how TLS certificate management can be performed, another option to look into is letting the AWS LB handle this.

For Confluence to be fully functional a database and a shared home volume must be stood up if you want to avoid any issues. If you intend on moving beyond evaluating Confluence then I seriously recommend getting these resources in place so as to avoid any data loss or time wasted re-building indexes. 

Dylan.

[Balazs Tarnok]

The last few days I was trying to deploy jira and confluence with different kind of configs but non of them were successful :((((

In my personal account now I have a classic load balancer (internet-facing scheme) with these listeners:

Load Balancer Protocol	Load Balancer Port	Instance Protocol	Instance Port	Cipher	SSL Certificate
TCP	443	TCP	31837	N/A	N/A
TCP	80	TCP	30784	N/A	N/A

This is what was created by your recommended helm command: 

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress

I added 'data: use-forwarded-headers: true' to the configmap and applied it. Stickiness options not available for TCP protocols.

I tried to skip the cert-manager and tls parts so in the values.yaml set 'ingress.https=false' and did not set tlsSecretName, the path is '/jira'.

I got these results with the dns of the lb:

with path /jira - HTTP ERROR 503

The pod is running, the service looks fine, here are the logs from the controller pod:

I0804 20:40:39.322143 7 controller.go:165] "Backend successfully reloaded"
I0804 20:40:39.322236 7 controller.go:176] "Initial sync, sleeping for 1 second"
I0804 20:40:39.322275 7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-nginx-controller-b65df6fbb-wwhcd", UID:"8fdbc470-f356-4145-87d5-f07af92fe3e5", APIVersion:"v1", ResourceVersion:"11838", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0804 20:48:02.441000 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"ingress-nginx-controller", UID:"d150abee-0f4f-45ec-8163-ff7d0a9d82df", APIVersion:"v1", ResourceVersion:"13473", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress/ingress-nginx-controller
I0804 20:48:02.443524 7 controller.go:148] "Configuration changes detected, backend reload required"
I0804 20:48:02.560310 7 controller.go:165] "Backend successfully reloaded"
I0804 20:48:02.560508 7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-nginx-controller-b65df6fbb-wwhcd", UID:"8fdbc470-f356-4145-87d5-f07af92fe3e5", APIVersion:"v1", ResourceVersion:"11838", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.14.217 - - [04/Aug/2021:20:48:38 +0000] "" 400 0 "-" "-" 0 0.270 [] [] - - - - 90c6712480ef7a2ed384538b5fd6b7d8
192.168.60.51 - - [04/Aug/2021:20:59:56 +0000] "GET / HTTP/1.1" 400 150 "-" "-" 18 0.051 [] [] - - - - d19d9b204c3523bdb48cd89204705391
192.168.60.51 - - [04/Aug/2021:21:03:43 +0000] "GET / HTTP/1.1" 400 150 "-" "-" 18 0.170 [] [] - - - - 7343e134dfc06624310427c1bb24d29b
192.168.60.51 - - [04/Aug/2021:21:12:14 +0000] "GET / HTTP/1.1" 400 150 "-" "-" 18 0.269 [] [] - - - - f65f6736388470c1044dcbfca8dcfb9b
W0804 21:19:04.829985 7 controller.go:992] Service "datacenter/jira-datacenter" does not have any active Endpoint.
I0804 21:19:04.872837 7 main.go:112] "successfully validated configuration, accepting" ingress="jira-datacenter/datacenter"
I0804 21:19:04.886927 7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"jira-datacenter", UID:"8b74da63-ee0a-4530-9665-62e20d3bd28e", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"20029", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0804 21:19:08.154482 7 controller.go:992] Service "datacenter/jira-datacenter" does not have any active Endpoint.
I0804 21:19:08.154684 7 controller.go:148] "Configuration changes detected, backend reload required"
I0804 21:19:08.243060 7 controller.go:165] "Backend successfully reloaded"
I0804 21:19:08.243422 7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-nginx-controller-b65df6fbb-wwhcd", UID:"8fdbc470-f356-4145-87d5-f07af92fe3e5", APIVersion:"v1", ResourceVersion:"11838", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0804 21:19:11.487821 7 controller.go:992] Service "datacenter/jira-datacenter" does not have any active Endpoint.
192.168.14.217 - - [04/Aug/2021:21:19:32 +0000] "GET /jira HTTP/1.1" 503 40 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" 494 0.044 [datacenter-jira-datacenter-80] [] 192.168.61.127:8080 40 0.044 503 3ec78d4929bb54b408d3769aceccefda
I0804 21:19:39.282324 7 status.go:284] "updating Ingress status" namespace="datacenter" ingress="jira-datacenter" currentValue=[] newValue=[{IP: Hostname:a3bb9a4f8d2254b15800ea34201983ba-1260621478.eu-west-1.elb.amazonaws.com Ports:[]}]
I0804 21:19:39.296313 7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"jira-datacenter", UID:"8b74da63-ee0a-4530-9665-62e20d3bd28e", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"20166", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
192.168.60.51 - - [04/Aug/2021:21:20:48 +0000] "GET / HTTP/1.1" 404 146 "-" "curl/7.61.1" 135 0.001 [upstream-default-backend] [] 127.0.0.1:8181 146 0.000 404 7fc6bcd42b1af0ad88827857c46eb81f
192.168.14.217 - - [04/Aug/2021:21:20:54 +0000] "GET /jira HTTP/1.1" 503 5 "-" "curl/7.61.1" 139 0.005 [datacenter-jira-datacenter-80] [] 192.168.61.127:8080 5 0.004 503 cd37d9a242688e04de399ad4625670ea

I spinned up a simple app (https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/) just to test the clb with another ingress, it works fine:

I also tried internal clb and alb with different configs in our organizational account but the one that I detailed above is the one that you recommended but it is still not working :(

I hope you can give me some advice and thanks for your help!

Regards,

Balazs

[Dylan Rathbone]

Hi Balazs,

would you mind passing on the full configuration of the ingress stanza within your values.yaml?

As an additional means of root causing the problem, you can eliminate the ingress from the mix and confirm that the service can actually be hit via the browser. This can be done with port-forwarding...

Get the Jira URL by running these commands in the same shell:
Step 1: Get the pod name
$ export POD_NAME=$(kubectl get pods --namespace jira -l "app.kubernetes.io/instance=jira" -o jsonpath="{.items[0].metadata.name}")
Step 2 (Optional): Check if the pod name has been exported successfully and the pod status
$ echo POD_NAME: $POD_NAME && echo POD_STATUS: $(kubectl get pod $POD_NAME -o jsonpath='{.status.phase}')
Step 3: Wait for pods up and running, then forward pod port to localhost
$ kubectl --namespace jira port-forward $POD_NAME 8080
Step 4: Access Jira on localhost:
http://localhost:8080

If you can confirm the service is active via the browser with this approach, it would indicate that there is definitely an issue with the Ingress setup which we can then further investigate.

Thanks.

[Balazs Tarnok]

Hi Dylan,

Thanks for your help and quick replies!

I tried to deploy jira, confluence, crowd without an ingress, in all cases got the same result. In step 2 I get this message 'Error from server (NotFound): pods "jira-datacenter-0" not found'. If I check the browser at localhost:8080 it says 'ERR_CONNECTION_REFUSED'. I did not find any errors in the logs of the pod.

-bash-4.2$ helm test jira-datacenter -n datacenter
Pod jira-datacenter-application-status-test pending
Pod jira-datacenter-application-status-test pending
Pod jira-datacenter-application-status-test pending
Pod jira-datacenter-application-status-test running
Pod jira-datacenter-application-status-test succeeded
Pod jira-datacenter-shared-home-permissions-test pending
Pod jira-datacenter-shared-home-permissions-test pending
Pod jira-datacenter-shared-home-permissions-test pending
Pod jira-datacenter-shared-home-permissions-test succeeded
NAME: jira-datacenter
LAST DEPLOYED: Thu Aug 5 06:29:04 2021
NAMESPACE: datacenter
STATUS: deployed
REVISION: 1
TEST SUITE: jira-datacenter-application-status-test
Last Started: Thu Aug 5 06:29:56 2021
Last Completed: Thu Aug 5 06:30:00 2021
Phase: Succeeded
TEST SUITE: jira-datacenter-shared-home-permissions-test
Last Started: Thu Aug 5 06:30:00 2021
Last Completed: Thu Aug 5 06:30:05 2021
Phase: Succeeded
NOTES:
Thank you for installing Jira.

Your release is named jira-datacenter, and resides in namespace datacenter.

Please run sanity tests against the release to verify it's healthy:

$ helm test jira-datacenter -n datacenter

If the Kubernetes resources in the release are still starting up, then the tests may fail, so it
is advisable to wait for the tests to pass before continuing.

To see the custom values you used for this release:

$ helm get values jira-datacenter -n datacenter

Get the Jira URL by running these commands in the same shell:
Step 1: Get the pod name
$ export POD_NAME=$(kubectl get pods --namespace datacenter -l "app.kubernetes.io/instance=jira-datacenter" -o jsonpath="{.items[0].metadata.name}")
Step 2 (Optional): Check if the pod name has been exported successfully and the pod status
$ echo POD_NAME: $POD_NAME && echo POD_STATUS: $(kubectl get pod $POD_NAME -o jsonpath='{.status.phase}')
Step 3: Wait for pods up and running, then forward pod port to localhost
$ kubectl --namespace datacenter port-forward $POD_NAME 8080
Step 4: Access Jira on localhost:
http://localhost:8080

#################################################################################
###### WARNING: Persistent volume is not used for Local Home!!! #####
###### Data will be lost when the pod is terminated. #####
#################################################################################

#################################################################################
###### WARNING: Persistent volume is not used for Shared Home!!! #####
###### Data will be lost when the pod is terminated. #####
#################################################################################

For further documentation, see https://github.com/atlassian-labs/data-center-helm-charts/tree/master/docs
-bash-4.2$ export POD_NAME=$(kubectl get pods --namespace datacenter -l "app.kubernetes.io/instance=jira-datacenter" -o jsonpath="{.items[0].metadata.name}")
-bash-4.2$ echo POD_NAME: $POD_NAME && echo POD_STATUS: $(kubectl get pod $POD_NAME -o jsonpath='{.status.phase}')
POD_NAME: jira-datacenter-0
Error from server (NotFound): pods "jira-datacenter-0" not found
POD_STATUS:
-bash-4.2$ kubectl --namespace datacenter port-forward $POD_NAME 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
^C-bash-4.2$ k logs -n datacenter jira-datacenter-0 | grep ERROR
-bash-4.2$ k logs -n datacenter jira-datacenter-0 | grep Error
-bash-4.2$ k logs -n datacenter jira-datacenter-0 | grep error
-bash-4.2$ k get -n datacenter events
LAST SEEN TYPE REASON OBJECT MESSAGE
7m51s Normal Scheduled pod/jira-datacenter-0 Successfully assigned datacenter/jira-datacenter-0 to ip-10-129-103-123.eu-west-1.compute.internal
7m51s Normal Pulling pod/jira-datacenter-0 Pulling image "atlassian/jira-software:8.13.9-jdk11"
7m33s Normal Pulled pod/jira-datacenter-0 Successfully pulled image "atlassian/jira-software:8.13.9-jdk11" in 17.930797725s
7m23s Normal Created pod/jira-datacenter-0 Created container jira
7m23s Normal Started pod/jira-datacenter-0 Started container jira
7m34s Normal Scheduled pod/jira-datacenter-application-status-test Successfully assigned datacenter/jira-datacenter-application-status-test to ip-10-129-103-12.eu-west-1.compute.inte rnal
7m34s Normal Pulling pod/jira-datacenter-application-status-test Pulling image "alpine"
7m33s Normal Pulled pod/jira-datacenter-application-status-test Successfully pulled image "alpine" in 1.746776338s
7m33s Normal Created pod/jira-datacenter-application-status-test Created container test
7m32s Normal Started pod/jira-datacenter-application-status-test Started container test
6m59s Normal Scheduled pod/jira-datacenter-application-status-test Successfully assigned datacenter/jira-datacenter-application-status-test to ip-10-129-103-12.eu-west-1.compute.inte rnal
6m59s Normal Pulled pod/jira-datacenter-application-status-test Container image "alpine" already present on machine
6m59s Normal Created pod/jira-datacenter-application-status-test Created container test
6m59s Normal Started pod/jira-datacenter-application-status-test Started container test
6m56s Normal Scheduled pod/jira-datacenter-shared-home-permissions-test Successfully assigned datacenter/jira-datacenter-shared-home-permissions-test to ip-10-129-103-12.eu-west-1.compute .internal
6m56s Normal Pulling pod/jira-datacenter-shared-home-permissions-test Pulling image "debian:stable-slim"
6m52s Normal Pulled pod/jira-datacenter-shared-home-permissions-test Successfully pulled image "debian:stable-slim" in 3.223335995s
6m52s Normal Created pod/jira-datacenter-shared-home-permissions-test Created container test
6m52s Normal Started pod/jira-datacenter-shared-home-permissions-test Started container test
6m51s Normal Killing pod/jira-datacenter-shared-home-permissions-test Stopping container test
7m52s Normal SuccessfulCreate statefulset/jira-datacenter create Pod jira-datacenter-0 in StatefulSet jira-datacenter successful
-bash-4.2$ k get -n datacenter ep
NAME ENDPOINTS AGE
jira-datacenter 100.64.67.179:8080 8m16s

[Balazs Tarnok]

If I try this same method with the simple 'apple' app then I got the same error after the second command and in the browser. But if I curl localhost:5678 then I get the reply 'apple'.

If I curl localhost:8080, I get an empty reply but it shows that handling connection:

-bash-4.2$ kubectl --namespace datacenter port-forward $POD_NAME 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080

[Dylan Rathbone]

Hi Balazs,

On the 27th July you had mentioned that you could in fact load the confluence setup page via the browser by using the external IP of the load balancer. Is this no longer the case? 

Can you also please supply the output for:

• kubectl get pods --namespace datacenter
• kubectl describe pods --namespace datacenter
• the values.yaml file you are using for deploying confluence.

Thanks.

[Balazs Tarnok]

Hi Dylan,

We could load the confluence setup page in my personal account where we do not have any restrictions for the load balancer.

ingress:
create: true
nginx: true
maxBodySize: 250m
host: <DNS name of the load balancer>
path: "/"
annotations: {}
https: true
tlsSecretName:

 Then we tried to use the recommended NGINX controller in our organizations's account, we had to add this custom.yaml to the load balancer setup:

controller:
service:
internal:
enabled: true
annotations:
"service.beta.kubernetes.io/aws-load-balancer-internal": "true"
"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "arn:aws:acm:eu-west-1:..."
"service.beta.kubernetes.io/aws-load-balancer-backend-protocol": https
"service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "443"
"service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags": "Environment=prd..."
"external-dns.alpha.kubernetes.io/hostname": "atlassian-ingress.wsteat.nl.aegon.io"
loadBalancerSourceRanges:
- 10.0.0.0/8
- 198.39.0.0/16
- 162.123.0.0/16
- 161.179.0.0/16
- 100.64.0.0/16

 With this setup we could not get to the confluence setup page. Usually we got HTTP 408 Error. 

Then we changed to the AWS load balancer controller https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/  and set it up with an application load balancer:

ingress:
create: true
nginx: false
maxBodySize: 250m
host: <Route53 Record name>
path: "/"
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:...
alb.ingress.kubernetes.io/subnets: subnet-..., subnet-...
alb.ingress.kubernetes.io/inbound-cidrs: >-
10.0.0.0/8,
198.39.0.0/16,
162.123.0.0/16,
161.179.0.0/16,
100.64.0.0/16
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
#alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled={{ .Values.logs.enabled }},access_logs.s3.bucket={{ .Values.logs.s3_bucket }},access_logs.s3.prefix={{ .Values.application }},idle_timeout.timeout_seconds={{ .Values.ingress.loadbalancer.timeout }}
alb.ingress.kubernetes.io/success-codes: 200,302
alb.ingress.kubernetes.io/tags: "Environment=prd,Division=agt,..."
external-dns.alpha.kubernetes.io/hostname: atlassian-ingress.wsteat.nl.aegon.io
https: true
tlsSecretName:

 With this setup we could get to the jira, crowd, bitbucket setup page. In confluence it caused a bit of problem that there is an ingress and an ingress-setup but we did not have the HTTP 408 error anymore. The disadvantage of this setup is that it creates a new load balancer for every ingress so that could be a further investigation to use only one load balancer for all the deployments because as I know it is the best practice (for example more cost effective)

• Do you recommend the NGINX ingress controller because that would not be specific for a cloud provider?
• The recommended helm nginx istallation setup a classic load balancer. As I know for new deployments in aws it is more recommended to use an alb or an nlb.
• Why did you prefer to have an ingress and an ingress setup for confluence?
• Based on these code blocks do you have any feeling what caused the problem with the classic load balancer in our organization's account?

This was just a PoC in our team, maybe we would work on production datacenter deployments at the beginning of the next year. 

Thanks,

Balazs

[Dylan Rathbone]

Hi Balazs,

Responses to your questions below:

• Do you recommend the NGINX ingress controller because that would not be specific for a cloud provider?
  
  • Response> We don't have any recommendations on this front, however, as you've stated, the charts were tailored with the NGINX ingress controller because of its agnostic nature. You are free to use any controller you want though, and it looks like you've had success with the AWS Load Balancer Controller.
• The recommended helm nginx istallation setup a classic load balancer. As I know for new deployments in aws it is more recommended to use an alb or an nlb.
  • Response> The load balancer provisioned certainly depends on your use case, a classic has the advantage of being able to serve HTTP & TCP traffic. However you are correct, an NLB is considered in most cases the recommended approach. By default when installing the NGINX controller an ELB (classic) will be used, however an NLB can also be provisioned, see here for details. Those instructions also detail how an ALB can be provisioned. 
• Why did you prefer to have an ingress and an ingress setup for confluence?
  • Response> Can you elaborate on this, I'm not sure what you mean?
• Based on these code blocks do you have any feeling what caused the problem with the classic load balancer in our organization's account?
  • Response> Balazs this is hard to say, certainly your environmental constraint's will definitely have input into how your ingress is configured. The topic of Ingress's and K8s is broad, with many competing solutions and configurations. The example documented in the helm chart repo serves as a reference for how an Ingress and associated Ingress object can be installed and configured, the caveat being, there may need to be additional config applied for things to work appropriately in the target environment.

As an FYI, your point on the AWS Load balancer controller spinning up a new load balancer for each ingress object is valid. This is something that the NGINX controller does not do, it can utilize a single LB for many Ingress objects across different namespaces. This is definitely something to consider for a production deployment.

Thanks,

Dylan.