no access to webpage after upgrade

Question

When upgrading from Jira Software from 8.15 to anything greater than 8.15, I can no longer access the web interface. We use https and I've

1. manually added the custom modifications to server.xml, web.xml, and verified jira-application.properties, and setenv.sh files.

2. used the config.sh to modify the web settings

3.  check the service jira status and system is "online"

No matter what I cannot access the web interface. I've looked through the catalina.out logs but nothing is jumping out and screaming "FAILED"

The SSL cert is valid. I've ran through every help page I could find that is relevant to my issue. Any help would be greatly appreciated.

unknown · Answer

am seeing (filtered to show only "START" labeled items) from the post-upgrade logs:

Line 4: [2023-02-17T13:01:40.163-0500][1.020s][info][gc,start ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
Line 17: [2023-02-17T13:01:41.900-0500][2.757s][info][gc,start ] GC(1) Pause Young (Concurrent Start) (Metadata GC Threshold)
Line 28: [2023-02-17T13:01:41.945-0500][2.802s][info][gc ] GC(1) Pause Young (Concurrent Start) (Metadata GC Threshold) 109M->21M(384M) 45.236ms
Line 42: [2023-02-17T13:01:41.952-0500][2.809s][info][gc,start ] GC(2) Pause Remark
Line 48: [2023-02-17T13:01:41.961-0500][2.818s][info][gc,start ] GC(2) Pause Cleanup
Line 54: [2023-02-17T13:01:42.200-0500][3.057s][info][gc,start ] GC(3) Pause Young (Normal) (G1 Evacuation Pause)
Line 67: [2023-02-17T13:01:43.457-0500][4.314s][info][gc,start ] GC(4) Pause Young (Normal) (G1 Evacuation Pause)
Line 80: [2023-02-17T13:01:44.693-0500][5.550s][info][gc,start ] GC(5) Pause Young (Normal) (G1 Evacuation Pause)
Line 93: [2023-02-17T13:01:45.873-0500][6.730s][info][gc,start ] GC(6) Pause Young (Normal) (G1 Evacuation Pause)
Line 106: [2023-02-17T13:01:47.693-0500][8.550s][info][gc,start ] GC(7) Pause Young (Normal) (G1 Evacuation Pause)
Line 119: [2023-02-17T13:01:48.592-0500][9.449s][info][gc,start ] GC(8) Pause Young (Normal) (G1 Evacuation Pause)
Line 132: [2023-02-17T13:01:49.524-0500][10.381s][info][gc,start ] GC(9) Pause Young (Normal) (G1 Evacuation Pause)
Line 145: [2023-02-17T13:01:50.692-0500][11.549s][info][gc,start ] GC(10) Pause Young (Concurrent Start) (Metadata GC Threshold)
Line 156: [2023-02-17T13:01:50.720-0500][11.577s][info][gc ] GC(10) Pause Young (Concurrent Start) (Metadata GC Threshold) 181M->36M(462M) 28.378ms
Line 170: [2023-02-17T13:01:50.739-0500][11.596s][info][gc,start ] GC(11) Pause Remark
Line 176: [2023-02-17T13:01:50.759-0500][11.616s][info][gc,start ] GC(11) Pause Cleanup
Line 182: [2023-02-17T13:01:54.269-0500][15.126s][info][gc,start ] GC(12) Pause Young (Concurrent Start) (System.gc())
Line 193: [2023-02-17T13:01:54.280-0500][15.137s][info][gc ] GC(12) Pause Young (Concurrent Start) (System.gc()) 290M->45M(462M) 11.778ms
Line 207: [2023-02-17T13:01:54.302-0500][15.159s][info][gc,start ] GC(13) Pause Remark
Line 213: [2023-02-17T13:01:54.318-0500][15.175s][info][gc,start ] GC(13) Pause Cleanup
Line 219: [2023-02-17T13:01:55.743-0500][16.600s][info][gc,start ] GC(14) Pause Young (Normal) (G1 Evacuation Pause)
Line 232: [2023-02-17T13:01:57.674-0500][18.530s][info][gc,start ] GC(15) Pause Young (Normal) (G1 Evacuation Pause)
Line 245: [2023-02-17T13:01:58.885-0500][19.742s][info][gc,start ] GC(16) Pause Young (Concurrent Start) (Metadata GC Threshold)
Line 256: [2023-02-17T13:01:58.920-0500][19.777s][info][gc ] GC(16) Pause Young (Concurrent Start) (Metadata GC Threshold) 232M->55M(556M) 34.973ms
Line 270: [2023-02-17T13:01:58.955-0500][19.812s][info][gc,start ] GC(17) Pause Remark
Line 276: [2023-02-17T13:01:58.976-0500][19.833s][info][gc,start ] GC(17) Pause Cleanup
Line 282: [2023-02-17T13:02:01.490-0500][22.347s][info][gc,start ] GC(18) Pause Young (Normal) (G1 Evacuation Pause)
Line 295: [2023-02-17T13:02:02.620-0500][23.477s][info][gc,start ] GC(19) Pause Young (Normal) (G1 Evacuation Pause)
Line 308: [2023-02-17T13:02:04.852-0500][25.709s][info][gc,start ] GC(20) Pause Young (Normal) (G1 Evacuation Pause)
Line 321: [2023-02-17T13:02:06.853-0500][27.710s][info][gc,start ] GC(21) Pause Young (Normal) (G1 Evacuation Pause)
Line 334: [2023-02-17T13:02:08.730-0500][29.587s][info][gc,start ] GC(22) Pause Young (Normal) (G1 Evacuation Pause)
Line 347: [2023-02-17T13:02:09.833-0500][30.690s][info][gc,start ] GC(23) Pause Young (Normal) (G1 Evacuation Pause)
Line 360: [2023-02-17T13:02:10.236-0500][31.093s][info][gc,start ] GC(24) Pause Young (Normal) (G1 Evacuation Pause)
Line 373: [2023-02-17T13:02:11.950-0500][32.807s][info][gc,start ] GC(25) Pause Young (Normal) (G1 Evacuation Pause)
Line 386: [2023-02-17T13:02:12.856-0500][33.713s][info][gc,start ] GC(26) Pause Young (Normal) (G1 Evacuation Pause)
Line 399: [2023-02-17T13:02:14.260-0500][35.117s][info][gc,start ] GC(27) Pause Young (Normal) (G1 Evacuation Pause)
Line 412: [2023-02-17T13:02:16.751-0500][37.608s][info][gc,start ] GC(28) Pause Young (Normal) (G1 Evacuation Pause)
Line 425: [2023-02-17T13:02:16.914-0500][37.771s][info][gc,start ] GC(29) Pause Young (Concurrent Start) (Metadata GC Threshold)
Line 436: [2023-02-17T13:02:16.956-0500][37.813s][info][gc ] GC(29) Pause Young (Concurrent Start) (Metadata GC Threshold) 277M->210M(802M) 41.823ms
Line 450: [2023-02-17T13:02:17.177-0500][38.034s][info][gc,start ] GC(30) Pause Remark
Line 456: [2023-02-17T13:02:17.323-0500][38.180s][info][gc,start ] GC(30) Pause Cleanup
Line 462: [2023-02-17T13:02:17.925-0500][38.782s][info][gc,start ] GC(31) Pause Young (Concurrent Start) (GCLocker Initiated GC)
Line 473: [2023-02-17T13:02:17.952-0500][38.809s][info][gc ] GC(31) Pause Young (Concurrent Start) (GCLocker Initiated GC) 545M->221M(964M) 26.601ms
Line 487: [2023-02-17T13:02:18.179-0500][39.036s][info][gc,start ] GC(32) Pause Remark
Line 493: [2023-02-17T13:02:18.377-0500][39.234s][info][gc,start ] GC(32) Pause Cleanup
Line 499: [2023-02-17T13:02:19.903-0500][40.760s][info][gc,start ] GC(33) Pause Young (Normal) (G1 Evacuation Pause)
Line 512: [2023-02-17T13:02:21.097-0500][41.954s][info][gc,start ] GC(34) Pause Young (Normal) (G1 Evacuation Pause)
Line 525: [2023-02-17T13:02:22.621-0500][43.478s][info][gc,start ] GC(35) Pause Young (Normal) (G1 Evacuation Pause)
Line 538: [2023-02-17T13:02:23.822-0500][44.679s][info][gc,start ] GC(36) Pause Young (Normal) (G1 Evacuation Pause)
Line 551: [2023-02-17T13:02:25.168-0500][46.024s][info][gc,start ] GC(37) Pause Young (Normal) (G1 Evacuation Pause)
Line 564: [2023-02-17T13:02:26.556-0500][47.412s][info][gc,start ] GC(38) Pause Young (Normal) (GCLocker Initiated GC)
Line 577: [2023-02-17T13:02:26.818-0500][47.675s][info][gc,start ] GC(39) Pause Young (Concurrent Start) (Metadata GC Threshold)
Line 588: [2023-02-17T13:02:26.851-0500][47.708s][info][gc ] GC(39) Pause Young (Concurrent Start) (Metadata GC Threshold) 465M->358M(1158M) 33.045ms
Line 602: [2023-02-17T13:02:27.343-0500][48.199s][info][gc,start ] GC(40) Pause Remark
Line 608: [2023-02-17T13:02:27.723-0500][48.580s][info][gc,start ] GC(40) Pause Cleanup
Line 614: [2023-02-17T13:02:28.198-0500][49.055s][info][gc,start ] GC(41) Pause Young (Concurrent Start) (G1 Evacuation Pause)
Line 625: [2023-02-17T13:02:28.246-0500][49.103s][info][gc ] GC(41) Pause Young (Concurrent Start) (G1 Evacuation Pause) 932M->380M(1158M) 47.976ms
Line 639: [2023-02-17T13:02:28.624-0500][49.481s][info][gc,start ] GC(42) Pause Remark
Line 645: [2023-02-17T13:02:28.887-0500][49.744s][info][gc,start ] GC(42) Pause Cleanup
Line 651: [2023-02-17T13:02:29.543-0500][50.400s][info][gc,start ] GC(43) Pause Young (Normal) (G1 Evacuation Pause)
Line 664: [2023-02-17T13:02:31.248-0500][52.105s][info][gc,start ] GC(44) Pause Young (Normal) (G1 Evacuation Pause)
Line 677: [2023-02-17T13:02:32.802-0500][53.659s][info][gc,start ] GC(45) Pause Young (Normal) (G1 Evacuation Pause)
Line 690: [2023-02-17T13:02:37.249-0500][58.106s][info][gc,start ] GC(46) Pause Young (Normal) (G1 Evacuation Pause)
Line 703: [2023-02-17T13:02:39.704-0500][60.560s][info][gc,start ] GC(47) Pause Young (Normal) (G1 Evacuation Pause)
Line 716: [2023-02-17T13:02:42.104-0500][62.961s][info][gc,start ] GC(48) Pause Young (Normal) (G1 Evacuation Pause)
Line 729: [2023-02-17T13:02:43.856-0500][64.713s][info][gc,start ] GC(49) Pause Young (Normal) (G1 Evacuation Pause)
Line 742: [2023-02-17T13:02:57.059-0500][77.916s][info][gc,start ] GC(50) Pause Young (Normal) (G1 Evacuation Pause)

unknown · Answer

Funny that the pre-update log file shows a ton of "Allocation Failures" and nothing similar to the other log file.

a small snippet:

2023-02-17T12:09:33.761-0500: 82310.505: [GC (Allocation Failure) [PSYoungGen: 241900K->11640K(238080K)] 1087109K->857081K(1252864K), 0.1659938 secs] [Times: user=0.56 sys=0.01, real=0.17 secs]
2023-02-17T12:18:21.199-0500: 82837.943: [GC (Allocation Failure) [PSYoungGen: 237432K->11339K(233472K)] 1082873K->857539K(1248256K), 0.1132079 secs] [Times: user=0.28 sys=0.00, real=0.11 secs]
2023-02-17T12:28:21.197-0500: 83437.942: [GC (Allocation Failure) [PSYoungGen: 233035K->10501K(229376K)] 1079235K->857094K(1244160K), 0.1052403 secs] [Times: user=0.25 sys=0.00, real=0.10 secs]
2023-02-17T12:34:58.438-0500: 83835.183: [GC (Allocation Failure) [PSYoungGen: 227600K->12470K(226304K)] 1074193K->859119K(1241088K), 0.1010076 secs] [Times: user=0.24 sys=0.00, real=0.10 secs]

Nic Brough -Adaptavist- · Answer

Yeah, Java application logs are always long and try to tell the developers the exact detail, which isn't what us admins really need when it goes bang.

The easiest way to read them is to ignore any line that is indented to the right.  Those lines are aimed at the developers and refer to specific lines of code, which isn't that much use to us.

The headlines that contain "error" are the most useful to us, although the "warnings" can tell us something sometimes as well.

There should be something in one of the left-aligned lines that says why it can't starty.