Confluence 5 to 6 Upgrade failure

[Robert Stout]

I get teh following error when starting the server

18-Jul-2017 15:16:15.157 SEVERE [http-nio-80-exec-8] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.OutOfMemoryError: Metaspace

I messed with the setenv.bat still no better

set CATALINA_OPTS=-Xms8192m -Xmx8192m -XX:+UseG1GC %CATALINA_OPTS%

Server 2012r2 4 proc 8gb mem 

Confluance 6.3.1

[Steven F Behnke]

Take a peek at this issue: https://jira.atlassian.com/browse/CONFSERVER-37925 

Set the value of -XX:MaxMetaspaceSize to something greater.

[Robert Stout]

Still get 

19-Jul-2017 08:31:29.264 SEVERE [http-nio-80-exec-9] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.OutOfMemoryError: Metaspace

Docs says that 6 does not need this setting.

[AnnWorley]

JIRA applications crash due to OutOfMemoryError Metaspace offers these suggestions:

• Remove the -XX:MaxMetaspaceSize parameter entirely, since Java 8 automatically increases the MetaSpace area as needed.
• Alternatively, increase the XX:MaxMetaspaceSize value.

Here are intructions to change the properties of the Windows service, either by command line or editing the registry.

Is it the case that you cannot start Confluence because of the metaspace error or you can start it but want to eliminate the error?

Nic mentioned Safe Mode, you could enable it and see if the errors are reduced or eliminated, however, I am not sure how long you would need to stay in safe mode to determine that a plugin is causing the issue because I am not sure how frequently you are seeing the error. Disabling or enabling all add-ons (using Safe Mode)

[AnnWorley]

Java 8, which is used with later Confluence versions, does not need the metaspace setting, and if it is present it can cause issues. 

Please see: Confluence crashes due to OutOfMemoryError Metaspace and Setting Properties for Windows Services.

 If you are starting Confluence as a service the settings in setenv.bat are ignored. If you have 8g RAM on the machine, please don't allocate all of it to Java as some needs to be available for the operating system.

[Robert Stout]

Yes it is starting as a service, so how do I get it to read the setenv? How do I allocate memory to java?

[Nic Brough -Adaptavist-]

The service won't read setenv, properties are set different places according to what you're running.  It's in the "Setting properties" doc Ann linked to

[Robert Stout]

Then what are the suggested changes to prevent the out of memory using this method?

[Nic Brough -Adaptavist-]

It's probably more complex than just settings.

The metaspace is a memory space Java loads classes into so it can use them.  In theory, it will drop most (but not all) classes out if they go unused long enough.  Running out of metaspace essentially means it's not able to drop enough classes out to let new ones in to be used.

You could do something like -XX:MaxMetaspaceSize=512m , but that could actually make it fall over faster, because the default metaspace maximum is "all the memory for the JVM".  It might also massively increase swapping (swapping is bad, it slows things down by 10s or 100s of times) or just lead to other out of memory errors in different parts of the memory as it chews up the space it can.

You could try increasing the overall memory for the JVM, but if you've got classes just chewing it up, all that will do is prolong it.

What you need to do is find out what classes are causing the problem.  A clean Confluence on its own is unlikely to be at fault because if it were, Atlassian would have been screamed at by so many clients, they'd have pulled the release.  It's far more likely to be an add-on or an unusual usage (for example, increasing the attachment limit to a couple of Gb and then using it.  I don't think that does actually break it now, but you see what I'm getting at)

I'd be looking at "safe mode" and the logs next, but, I've just typed all of that and suddenly realised there may be another problem.

In your question, you set the minimum and maximum memory for the JVM to 8192Mb.  Your server has 8Gb available.  You do need to allow the OS some working space!  A general rule - make sure the service settings set the maximum memory to at least 2Gb LESS than the server physical memory

[Robert Stout]

I followed the instructions from the link Setting Properties for Windows Services and found that the -XX:MaxMetaspaceSize setting was in the "Java Options" so I removed it and it is now working properly.

Now on to fixing the SSL config that broke during the upgrade.