Deploy Jira Deployment Openshift Kustomize

[Rekha Khandhadia]

I am new to deploying Jira.  I am using Kustomize to deploy Jira Software to Openshift Cluster.   I am following deployment example https://github.com/zimmertr/TKS-Deploy_Kubernetes_Apps/tree/master/Jira/overlays/example. 

• Database is separately deployed PostgreSQL in same namespace and created a jiradb database and user. 
• Downloaded the image jira-software:8.13.11-ubuntu-jdk11
• Part of this deployment 
  • Created Service 
  • PVC is deployed. 
• The issue is that we cannot change permissions nor run as user our security context is locked down.  
• Below is the deployment and error message 

containers:
- name: jira
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
securityContext:
#runAsUser: 2001
fsroup: 2001
image: corporate-registry/crunchydata/jira-software:8.13.11-ubuntu-jdk11
envFrom:
- configMapRef:
name: jira-env-vars
- secretRef:
name: jira-postgres-password
ports:
- name: "http-web"
containerPort: 8080
protocol: TCP
volumeMounts:
- name: jiradata
mountPath: /var/atlassian/application-data/jira
env:
- name: SET_PERMISSIONS
value: "false"
# resources:
# limits:
# cpu: 500m
# memory: 1Gi

volumes:
- name: jiradata
persistentVolumeClaim:
claimName: jiradata
serviceAccount: XXXXX
serviceAccountName: XXXX
terminationGracePeriodSeconds: 30

• Pod is failing with below error message in the log

INFO:root:Generating /etc/container_id from template container_id.j2
WARNING:root:Container not started as root. Bootstrapping skipped for '/etc/container_id'
INFO:root:Generating /opt/atlassian/jira/conf/server.xml from template server.xml.j2
WARNING:root:Container not started as root. Bootstrapping skipped for '/opt/atlassian/jira/conf/server.xml'
INFO:root:Generating /opt/atlassian/jira/atlassian-jira/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2
WARNING:root:Container not started as root. Bootstrapping skipped for '/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/seraph-config.xml'
INFO:root:/var/atlassian/application-data/jira/dbconfig.xml exists; skipping.
WARNING:root:Unsetting environment var JIRA_PASSWORD
WARNING:root:Unsetting environment var ATL_JDBC_PASSWORD
INFO:root:Running Jira with command '/opt/atlassian/jira/bin/start-jira.sh', arguments ['/opt/atlassian/jira/bin/start-jira.sh', '-fg']
executing as current user

`sMMMMMMMMMMMMMM+
MMMMMMMMMMMMMM
:sdMMMMMMMMMMM
MMMMMM
`sMMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sMMMMMMMMMMM MMMMM
MMMMMM `UOJ
`sMMMMMMMMMMMMM+ MMMMMM
MMMMMMMMMMMMMM +MMMMM
:sdMMMMMMMMMM MMMMM
MMMMMM `UOJ
MMMMMM
+MMMMM
MMMMM
`UOJ

Atlassian Jira
Version : 8.13.11

If you encounter issues starting or stopping Jira, please see the Troubleshooting guide at https://docs.atlassian.com/jira/jadm-docs-0813/Troubleshooting+installation

Using JIRA_HOME: /var/atlassian/application-data/jira

Server startup logs are located in /opt/atlassian/jira/logs/catalina.out
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.001s][error][logging] Error opening log file '/opt/atlassian/jira/logs/atlassian-jira-gc-2022-03-15_19-47-15.log': Permission denied
[0.001s][error][logging] Initialization of output 'file=/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log' using options 'filecount=5,filesize=20M' failed.
Invalid -Xlog option '-Xlog:gc*:file=/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log:tags,time,uptime,level:filecount=5,filesize=20M', see error log for details.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

[Yevhen · Accepted solution]

@Rekha Khandhadia OpenShift runs pods with a non privileged user having UID from a predefined range. When Jira starts, it cannot write a log file. There are a couple of fixes here:

* run as root (i.e. allow anyuid scc for your service account)

* build a custom image and make /opt/atlassian/jira writable for root group (unprivileged user belongs to this group)

* declare a runtime volume for /opt/jira/atlassian/logs (it should be emptyDir)

Either option should work. Perhaps, declaring a volume is the easiest.

[Rekha Khandhadia]

For declare runtime volume 

volumeMounts:
- name: jiradata
mountPath: /var/atlassian/application-data/jira
- name: logs
mountPath: /opt/jira/atlassian/logs

volumes:
- name: jiradata
persistentVolumeClaim:
claimName: jiradata
- name: logs
emptyDir: {}

I got same error ,  way I have define above correct. 

[0.001s][error][logging] Error opening log file '/opt/atlassian/jira/logs/atlassian-jira-gc-2022-03-16_19-15-36.log': Permission denied

[Yevhen]

@Rekha Khandhadia wrong mountPath? It should be /opt/atlassian/jira/logs not /opt/jira/atlassian/logs

[Rekha Khandhadia]

Thank you very much that worked.   For shared-home and local-home I have create below mapping,  please validate below specification.  PVC's are provisioned,  I should see a dir shared-home and local-home right?  

 volumeMounts:
- name: jiradata
mountPath: /var/atlassian/application-data/jira
- name: logs
mountPath: /opt/atlassian/jira/logs
- name: local-home
mountPath: /local-home
- name: shared-home
mountPath: /shared-home
env:
- name: SET_PERMISSIONS
value: "false"
volumes:
- name: jiradata
persistentVolumeClaim:
claimName: jiradata
- name: logs
emptyDir: {}
- name: local-home
persistentVolumeClaim:
claimName: local-home
- name: shared-home
persistentVolumeClaim:
claimName: shared-home

[Yevhen]

@Rekha Khandhadia Local home should be /var/atlassian/application-data/jira and shared home /var/atlassian/application-data/shared-home

[Rekha Khandhadia]

So we are mapping local-home and jiradata to same  /var/atlassian/application-data/jira,  I am confused do we need local-home and shared-home.

[Yevhen]

@Rekha Khandhadia why the same?

• local-home:  /var/atlassian/application-data/jira
• shared-home:/var/atlassian/application-data/shared-home

Two different mountPaths.

[Rekha Khandhadia]

I have jiradata to which is same as local-home,  so do we need jiradata

- name: jiradata
mountPath: /var/atlassian/application-data/jira

[Yevhen]

Jira data is your local home.

[Rekha Khandhadia]

I am still seeing this

error org.apache.catalina.core.StandardContext.postWorkDirectory Failed to create work directory [/opt/atlassian/jira/work/Catalina/localhost/ROOT] for context []

I also see 

22-Mar-2022 19:22:33.963 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 22-Mar-2022 19:22:33.970 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read 22-Mar-2022 19:22:33.981 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3254 ms 2022-03-22 19:22:34,079+0000 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira/dbconfig.xml 2022-03-22 19:22:34,118+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Trying to get encrypted password from xml and decrypt it 2022-03-22 19:22:34,119+0000 JIRA-Bootstrap INFO [c.a.j.config.database.DatabaseConfigHandler] Database password decryption not performed. 2022-03-22 19:22:34,359+0000 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running Jira startup checks. 2022-03-22 19:22:34,359+0000 JIRA-Bootstrap FATAL [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked. 2022-03-22 19:22:34,414+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Startup is complete. Jira is ready to serve. 2022-03-22 19:22:34,416+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage: --------------------------------------------------------------------------------- Heap memory : Used: 53 MiB. Committed: 615 MiB. Max: 6144 MiB Non-heap memory : Used: 49 MiB. Committed: 107 MiB. Max: 1536 MiB --------------------------------------------------------------------------------- TOTAL : Used: 102 MiB. Committed: 722 MiB. Max: 7680 MiB ---------------------------------------------------------------------------------

I am not able to access JIRA

[Yevhen]

@Rekha Khandhadia regarding database encryption error, I have found a similar thread https://community.atlassian.com/t5/Jira-questions/Jira-won-t-start-after-hard-reboot-due-to-password-decryption/qaq-p/1224018 Have you tried flushing everything, including PVCs, and running again? Also, it's worth checking if there are multiple Jira instances running and using the same directory. See: https://community.atlassian.com/t5/Jira-Core-Server-questions/IRA-Locked-Error-Won-t-startup/qaq-p/1445191

As to error org.apache.catalina.core.StandardContext.postWorkDirectory Failed to create work directory [/opt/atlassian/jira/work/Catalina/localhost/ROOT] for context [] I think you can disregard it or add yet another runtime volume for this directory.

[Rekha Khandhadia]

I finally figured out that we need to set_permission=true and configured the volumes as suggested.  I think the JIRA is up but when i try to access via route I see below error in the log 

2022-03-23 20:59:30,872+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Startup is complete. Jira is ready to serve. 2022-03-23 20:59:30,874+0000 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage: --------------------------------------------------------------------------------- Heap memory : Used: 48 MiB. Committed: 615 MiB. Max: 6144 MiB Non-heap memory : Used: 49 MiB. Committed: 107 MiB. Max: 1536 MiB --------------------------------------------------------------------------------- TOTAL : Used: 96 MiB. Committed: 722 MiB. Max: 7680 MiB --------------------------------------------------------------------------------- 23-Mar-2022 21:00:48.493 INFO [http-nio-8080-exec-2] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x020x000x010x000x010xfc0x030x03'<&xQ0xd0m0xb6?0xf40x811!Rk0xa3=0x0e"0xd9ec0xa90x9a0xd100x0dm0xb10xf7a0x9d]. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:431) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:503) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) 23-Mar-2022 21:00:48.493 INFO [http-nio-8080-exec-1] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x020x000x010x000x010xfc0x030x03?"0x180xd6B0xa7}0x950xe10xda0x930xcbM0xf9d0x1d0xe40x17Y0xfb0x0c0xdc\00xe8!0xfe0xb4ov0x820xe9]. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:431) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:503) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829)

I found some ticket that says I have to enable SSL in servers.xml but I am not sure how to do it via Kustomize

[Yevhen]

@Rekha Khandhadia  can you please share your route yaml? It looks like you reencrypt the request or passthough. In other words, you make https request to the backend which expects http (Jira server in this case).

[Rekha Khandhadia]

apiVersion: route.openshift.io/v1
kind: Route
metadata:
labels:
app: jira
name: jira
spec:
port:
targetPort: 8080
tls:
termination: edge
to:
kind: Service
name: jira
weight: 100
wildcardPolicy: None

[Rekha Khandhadia]

My database connection was not working once I fixed it and modified the route to edge.  I see the application is up.  Thank you very much for helping,  really appreciate it. 

[Yevhen]

@Rekha Khandhadia  jdbc url has nothing to do with the app route. Yes, edge is the way to go, so that tls termination happens on the OpenShift router level.

So, Jira is finally up and running?

[Rekha Khandhadia]

Yes, it is I have not gone any further than that.   This is part of JIRA migration.