dev environment fail

[SherazadeC]

Hello all,

I've been working on this for a bit and already posted here so sorry for the repetition. In summary, I've cloned a production environment to a different physical server to create a development environment. 

I've managed to get confluence to start but it does so with errors. 

I had this error first : 

BootstrapException: Unable to bootstrap application: Failed to parse config file: Error on line 33 of document : The content of elements must consist of well-formed character data or markup. Nested exception: The content of elements must consist of well-formed character data or markup.

Up until that point, confluence was still displayed in the browser with that error. I  tested some changes to the setenv.sh file (that didn't work), and then reverted back my changes and restarted confluence, now it starts with a 404 error. 

[root@hostname conf]# wget http://hostname:8090/
--2019-11-27 18:25:23-- http://hostname:8090/
Resolving <hostname> (<hostname>)... <ip address>
Connecting to <hostname> (<hostname>)|<ip address>|:8090... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-11-27 18:25:23 ERROR 404: Not Found.

[root@hostname confluence]# systemctl status -l confluence.service
● confluence.service - Confluence
Loaded: loaded (/usr/lib/systemd/system/confluence.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-11-27 17:41:06 CET; 2s ago
Process: 27131 ExecStop=/opt/atlassian/confluence/bin/stop-confluence.sh (code=exited, status=0/SUCCESS)
Process: 27203 ExecStart=/opt/atlassian/confluence/bin/start-confluence.sh (code=exited, status=0/SUCCESS)

Main PID: 27248 (java)
CGroup: /system.slice/confluence.service
└─27248 /opt/atlassian/confluence/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls .ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Datlassian.mail.senddisabled=true -Datlassian.notifications.disabled=true -Dconfluence.context.path= -Datlassian.plugins.startup .options= -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercen t=20 -Xloggc:/opt/atlassian/confluence/logs/gc-2019-11-27_17-41-06.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -Dignore.endorsed.dirs= -classpath /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava .io.tmpdir=/opt/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start

Nov 27 17:41:05 hostname systemd[1]: Starting Confluence...
Nov 27 17:41:05 hostname start-confluence.sh[27203]: To run Confluence in the foreground, start the server with start-confluence.sh -fg
Nov 27 17:41:05 hostname start-confluence.sh[27203]: executing as current user

Nov 27 17:41:05 hostname start-confluence.sh[27203]: If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Inst allation+Guide
Nov 27 17:41:05 hostname start-confluence.sh[27203]: Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

Nov 27 17:41:05 hostname start-confluence.sh[27203]: ---------------------------------------------------------------------------
Nov 27 17:41:05 hostname start-confluence.sh[27203]: Using Java: /opt/atlassian/confluence/jre//bin/java
Nov 27 17:41:06 hostname start-confluence.sh[27203]: 2019-11-27 17:41:06,269 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/syn chrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
Nov 27 17:41:06 hostname start-confluence.sh[27203]: ---------------------------------------------------------------------------
Nov 27 17:41:06 hostname systemd[1]: Started Confluence.

So apparently it looks for java in /jre//bin/java with a double slash which I don't understand why. 

Here is my setenv.sh file : 

# See the CATALINA_OPTS below for tuning the JVM arguments used to start Confluence.

echo "If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide"

# set the location of the pid file
if [ -z "$CATALINA_PID" ] ; then
if [ -n "$CATALINA_BASE" ] ; then
CATALINA_PID="$CATALINA_BASE"/work/catalina.pid
elif [ -n "$CATALINA_HOME" ] ; then
CATALINA_PID="$CATALINA_HOME"/work/catalina.pid
fi
fi
export CATALINA_PID

PRGDIR=`dirname "$0"`
if [ -z "$CATALINA_BASE" ]; then
if [ -z "$CATALINA_HOME" ]; then
LOGBASE=$PRGDIR
LOGTAIL=..
else
LOGBASE=$CATALINA_HOME
LOGTAIL=.
fi
else
LOGBASE=$CATALINA_BASE
LOGTAIL=.
fi

PUSHED_DIR=`pwd`
cd $LOGBASE
cd $LOGTAIL
LOGBASEABS=`pwd`
cd $PUSHED_DIR

echo ""
echo "Server startup logs are located in $LOGBASEABS/logs/catalina.out"
# IMPORTANT NOTE: Only set JAVA_HOME or JRE_HOME above this line
# Get standard Java environment variables
if $os400; then
# -r will Only work on the os400 if the files are:
# 1. owned by the user
# 2. owned by the PRIMARY group of the user
# this will not work if the user belongs in secondary groups
. "$CATALINA_HOME"/bin/setjre.sh
else
if [ -r "$CATALINA_HOME"/bin/setjre.sh ]; then
. "$CATALINA_HOME"/bin/setjre.sh
else
echo "Cannot find $CATALINA_HOME/bin/setjre.sh"
echo "This file is needed to run this program"
exit 1
fi
fi

echo "---------------------------------------------------------------------------"
echo "Using Java: $JRE_HOME/bin/java"

CONFLUENCE_CONTEXT_PATH=`$JRE_HOME/bin/java -jar $CATALINA_HOME/bin/confluence-context-path-extractor.jar $CATALINA_HOME`
export CONFLUENCE_CONTEXT_PATH
$JRE_HOME/bin/java -jar $CATALINA_HOME/bin/synchrony-proxy-watchdog.jar $CATALINA_HOME
echo "---------------------------------------------------------------------------"

# Set the JVM arguments used to start Confluence. For a description of the options, see
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx1024m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.notifications.disabled=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.mail.senddisabled=true ${CATALINA_OPTS}"


export CATALINA_OPTS

 Here are the error logs : 

2019-11-27 17:41:04,963 FATAL [localhost-startStop-3] [atlassian.spring.container.SpringContainerContext] getComponent Spring Application context has not been set
2019-11-27 17:41:04,963 WARN [localhost-startStop-3] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextDestroyed Failed to shut down plugin system during ServletContext shutdown: Spring Application context has not been$
2019-11-27 17:41:04,988 INFO [localhost-startStop-3] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence
27-Nov-2019 17:41:05.056 WARNING [localhost-startStop-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web$
27-Nov-2019 17:41:05.065 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8090"]
27-Nov-2019 17:41:05.077 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8090"]
27-Nov-2019 17:41:06.526 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.577 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.584 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.588 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.616 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.636 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
27-Nov-2019 17:41:06.896 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
27-Nov-2019 17:41:06.914 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
27-Nov-2019 17:41:06.919 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 457 ms
27-Nov-2019 17:41:06.926 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
27-Nov-2019 17:41:06.926 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.50
27-Nov-2019 17:41:08.922 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@511209a0]
27-Nov-2019 17:41:09.244 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
27-Nov-2019 17:41:09.245 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
27-Nov-2019 17:41:09.250 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Wed No$
27-Nov-2019 17:41:09.419 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSoc$
2019-11-27 17:41:09,855 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.7.2 [build 7601 based on commit hash 2b9cef12215b66d36aac7298ee6598161e5b8cbb] - synchrony version 2.1.0-$
27-Nov-2019 17:41:09.975 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
27-Nov-2019 17:41:10.044 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/v1/bayeux-sync1] onto handler of type [class org.springframework.web.socket.se$
27-Nov-2019 17:41:10.506 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServ$
27-Nov-2019 17:41:10.689 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispa$
27-Nov-2019 17:41:10.867 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult $
27-Nov-2019 17:41:10.868 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com$
27-Nov-2019 17:41:11.037 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
27-Nov-2019 17:41:11.094 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1849 ms
2019-11-27 17:41:13,594 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.DefaultAtlassianBootstrapManager] init Home is not configured properly:
com.atlassian.config.ConfigurationException: Failed to parse config file: Error on line 33 of document : The content of elements must consist of well-formed character data or markup. Nested exception: The content of elements must consi$
at com.atlassian.config.xml.DefaultDom4jXmlConfigurationPersister.load(DefaultDom4jXmlConfigurationPersister.java:35)
at com.atlassian.config.xml.DefaultDom4jXmlConfigurationPersister.load(DefaultDom4jXmlConfigurationPersister.java:65)
at com.atlassian.config.ApplicationConfig.load(ApplicationConfig.java:309)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:72)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:179)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:130)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:64)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4861)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.dom4j.DocumentException: Error on line 33 of document : The content of elements must consist of well-formed character data or markup. Nested exception: The content of elements must consist of well-formed character data o$
at org.dom4j.io.SAXReader.read(SAXReader.java:482)
at org.dom4j.io.SAXReader.read(SAXReader.java:343)
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.loadDocument(AbstractDom4jXmlConfigurationPersister.java:69)
at com.atlassian.config.xml.DefaultDom4jXmlConfigurationPersister.load(DefaultDom4jXmlConfigurationPersister.java:31)
... 16 more
2019-11-27 17:41:13,613 ERROR [localhost-startStop-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
Unable to bootstrap application: Failed to parse config file: Error on line 33 of document : The content of elements must consist of well-formed character data or markup. Nested exception: The content of elements must consist of well-f$
com.atlassian.config.bootstrap.BootstrapException: Unable to bootstrap application: Failed to parse config file: Error on line 33 of document : The content of elements must consist of well-formed character data or markup. Nested except$
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:39)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:130)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:64)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4861)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-11-27 17:41:14,290 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
27-Nov-2019 17:41:14.305 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
27-Nov-2019 17:41:14.313 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 7394 ms

 Thanks a for for your help

[Jakob KN]

A bit of a late update, but we've just experience the same error as you when copying Confluence production to a dev server "The content of elements must consist of well-formed character..."

In our case this was caused by our PostgreSQL user in the DB having a password containing the character "<" which caused the issue. 

Hopefully this is useful, in case someone comes across this post as well.

[Shannon S]

Hello @SherazadeC,

Thank you for bringing this to us. I would focus on the Spring Application Context Has Not Been Set error.

There are a set of steps you can take to narrow-down this issue:

• Confluence does not start due to Spring Application context has not been set 

Even though your exact cause might not be there, it will be related to something in that article.

If you have any trouble or questions, please let me know.

Regards,

Shannon