nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Having issues with installation

Balazs Tarnok

Hey All!

First of all, thank you for working on this great, new deployment option! In my organization we are also trying to go through the installation process but faced a few issues. It would be also really nice if we could have a call sometime, if it is possible.

We are using EKS on AWS, I just have a bit of experience with these technologies so maybe going to have a few basic questions too.

We tried to use different nginx installations: the one with helm in your documentation (classic lb) and two others from the original site (alb,nlb) (https://kubernetes.github.io/ingress-nginx/deploy/#aws). We realized that all these installations are creating internet-facing load balancers which is forbidden in our organization. My knowledge is basic about networking so I would like to know that would it cause any problem or further changes in other parts of your recommended installation if we would use an internal load balancer?

I also tried to install into my personal aws account where internet-facing load balancers are allowed. I installed a classic load balancer with the recommanded helm commands from here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/examples/ingress/INGRESS_NGINX.md. Then continued with the installation guide from here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/INSTALLATION.md. Skipped step 3,4,5 just to have the most basic installation. A pod, service and statefulset was created but the pod stucks in pending status. Is step 3,4,5 required to have a running pod?

Then I tried to deploy the ingress (step 4) too, changed values.yaml but the pod is still in pending status, the nginx controller 'could not find any active endpoint'. Is it a must have to set the 'use-forwarded-headers=true' in the configmap before the nginx controller installation? As it states here: https://github.com/atlassian-labs/data-center-helm-charts/blob/master/docs/CONFIGURATION.md

As I mentioned at the beginning, I have many questions, maybe even really basic ones but thank you for your help in advance, it would be great to have a discussion with you!

Best regards,

Balazs

Comments

Dylan Rathbone

Hi Balazs,

Thanks for showing an interest in the DC Helm charts! We can certainly provide some pointers that might assist in getting you moving along. First thing I recommend is focussing on the pod and its pending state, steps 3,4,5 are definitely not needed for kicking the tyres and evaluating a DC product provisioned via the charts.

By the way what product are you trying to install?

Back to the pending pod, I'd recommend using kubectl describe to get some lower level details as to why your pod is pending i.e.

kubectl describe pod <pod-name> -n <namespace>

Hopefully the events listing will shed some light on the problem. You can also use this official guide for some handy tips on debugging deployments.

Let's get these details first and work from there thumbs-up

As for provisioning an internal load balancer with ingress-nginx this can be done by supplying custom config when deploying. I'd recommend having a read through the offical readme for some more detail. What you're after I believe is something like this:

controller:
  service:
    internal:
      enabled: true
annotations:
      "service.beta.kubernetes.io/aws-load-balancer-internal": "true"

and then use this config as follows:

helm install ingress ingress-nginx/ingress-nginx --values custom-config.yaml

Balazs Tarnok

Thank you very much for your quick reply Dylan!

I could figure out that the pods were pending because there were not suffiecient memory and cpu so had to change the instance type of the nodes. In my personal account for developing I am trying to install confluence. But in our organization account we would like to deploy confluence, jira and crowd. We have there two m5.4xlarge type nodes. Do you have any recommended node type for our use case with these 3 applications?

In my account now I have a running pod with some log4j errors:

log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".

I created the ingress, there are also endpoints:

confluence-datacenter 192.168.26.123:8090,192.168.26.123:5701

These are the event logs

37m Normal Scheduled pod/confluence-datacenter-0 Successfully assigned datacenter/confluence-datacenter-0 to ip-192-168-30-206.eu-west-1.compute.internal
37m Normal Pulling pod/confluence-datacenter-0 Pulling image "atlassian/confluence-server:7.12.3-jdk11"
36m Normal Pulled pod/confluence-datacenter-0 Successfully pulled image "atlassian/confluence-server:7.12.3-jdk11" in 10.906763879s
36m Normal Created pod/confluence-datacenter-0 Created container confluence
36m Normal Started pod/confluence-datacenter-0 Started container confluence
36m Warning Unhealthy pod/confluence-datacenter-0 Readiness probe failed: Get "http://192.168.26.123:8090/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
36m Normal Sync ingress/confluence-datacenter-setup Scheduled for sync
36m Normal Sync ingress/confluence-datacenter Scheduled for sync
37m Normal SuccessfulCreate statefulset/confluence-datacenter create Pod confluence-datacenter-0 in StatefulSet confluence-datacenter successfu

I created a route53 record that would route traffic to the classic load balancer: dualstack.a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com. The record name is: confluence.datacenter.proba.com so in the values. yaml I set the ingress.host to this value.

When I try to load this page I get: ERR_NAME_NOT_RESOLVED

Here is the nginx controller pod log:

W0727 09:46:40.206676 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:40.245260 6 main.go:112] "successfully validated configuration, accepting" ingress="confluence-datacenter/datacenter"
I0727 09:46:40.270813 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter", UID:"d080fa51-02e6-4a08-afe5-72e0d33e0de9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7120", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0727 09:46:40.315199 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:40.362342 6 main.go:112] "successfully validated configuration, accepting" ingress="confluence-datacenter-setup/datacenter"
I0727 09:46:40.380022 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter-setup", UID:"9fffb04c-a000-4bcb-9382-29a9e38301fb", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7125", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0727 09:46:43.519960 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:46:43.523220 6 controller.go:146] "Configuration changes detected, backend reload required"
I0727 09:46:43.610188 6 controller.go:163] "Backend successfully reloaded"
I0727 09:46:43.610434 6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress", Name:"ingress-nginx-controller-d9458694b-rtzgf", UID:"5808b0ba-331a-4d18-91ad-3de3d86704bd", APIVersion:"v1", ResourceVersion:"3392", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0727 09:46:46.853322 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
W0727 09:47:01.934715 6 controller.go:981] Service "datacenter/confluence-datacenter" does not have any active Endpoint.
I0727 09:47:40.159965 6 status.go:284] "updating Ingress status" namespace="datacenter" ingress="confluence-datacenter" currentValue=[] newValue=[{IP: Hostname:a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com Ports:[]}]
I0727 09:47:40.159965 6 status.go:284] "updating Ingress status" namespace="datacenter" ingress="confluence-datacenter-setup" currentValue=[] newValue=[{IP: Hostname:a4292c3ec12b846f2a419c0b985625c2-1949832872.eu-west-1.elb.amazonaws.com Ports:[]}]
I0727 09:47:40.185936 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter-setup", UID:"9fffb04c-a000-4bcb-9382-29a9e38301fb", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7358", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0727 09:47:40.186885 6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"datacenter", Name:"confluence-datacenter", UID:"d080fa51-02e6-4a08-afe5-72e0d33e0de9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7359", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

Balazs Tarnok

I could get a bit further:
If I use the external ip of the load balancer in values.yaml at ingress.host then I can load the confluence setup page in a browser with the external ip. The browser still showing that it is not secure, I am a bit confused with the cert-manager and tls part.

Based on this I am having a problem with my route53 record and dns registration.

Dylan Rathbone

Hey Balazs,

Glad to hear you got your pods provisioned, I had a suspicion that K8s might have been having scheduling issues due to insufficient resources. Getting you're Pods scheduled is a good start, you're on your way now! wink . Just as a side note, you can tune the requested cpu and memory amounts required by each Pod. Doing this may have allowed you to provision Confluence to your existing nodes after all, see here for details.

With regards recommendations for the type of EC2 instance to use with Confluence, please take a look at this guide it should help give you an idea of what to use based on data sizes.

Balazs, it sounds like you are now using an external load balancer? If you want to externalize the Confluence service from the cluster then an external load balancer will be needed. Your DNS issues could be related to a number of things, namely pending verification by AWS of the domain name itself i.e. ".proba.com". If this has not yet been verified then it could explain why you can hit the service on the external IP of the LB but not the record "confluence.datacenter.proba.com". Take a look at the dig command for verifying the domain name.

For certificate issuance and management, when used in conjunction with the Ingress example we supply, we have had success with cert-manager and Let'sEncrypt. This setup must be in place before installing Confluence with Helm. What happens is on Confluence install a once off pod is created by the cert-issuer which then requests a TLS cert from Let'sEncrypt to associate with the Confluence service via the Ingress. This is only an example of how TLS certificate management can be performed, another option to look into is letting the AWS LB handle this.

For Confluence to be fully functional a database and a shared home volume must be stood up if you want to avoid any issues. If you intend on moving beyond evaluating Confluence then I seriously recommend getting these resources in place so as to avoid any data loss or time wasted re-building indexes.