Application cannot be null at /login.vm

[Taaniel-Martinus Paal]

Yesterday night, our confluence instance just decided to give up and went haywire. We do have a backup of the filesystem available but not for the database (migration underway and other priorities) so no way to restore to an earlier date. Below are logs from the night - everything was ok until 03:XX, then a long break in logs until 13:XX when confluence startup suddenly happened. After startup, it almost immediately failed again:

2023-11-07 13:32:45,924 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [host.util.concurrent.LockedTaskExecutor] executeUnderLock Failed whilst getting last modified date
java.util.concurrent.ExecutionException: org.eclipse.gemini.blueprint.service.importer.ServiceProxyDestroyedException: service proxy has been destroyed
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492)
...

Then, some weird errors and warnings:

2023-11-07 13:32:45,941 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.event.events.plugin.AsyncPluginFrameworkStartedEvent[source=com.atlassian.confluence.plugin.ConfluencePluginManager@4e18de6e]: service proxy has been destroyed
2023-11-07 13:32:45,964 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@19490b6: service proxy has been destroyed
2023-11-07 13:32:45,972 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@3448242d: service proxy has been destroyed
2023-11-07 13:32:45,976 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@2a77eb7d: service proxy has been destroyed
2023-11-07 13:32:45,980 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@691f6367: service proxy has been destroyed
2023-11-07 13:32:45,986 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@10dd48df: service proxy has been destroyed
2023-11-07 13:32:45,990 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@355807e4: service proxy has been destroyed
2023-11-07 13:32:45,997 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@2f38d652: service proxy has been destroyed
2023-11-07 13:32:46,003 ERROR [AtlassianEvent::CustomizableThreadFactory-1] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.confluence.util.i18n.event.I18NCacheInitEvent@3cec8a59: service proxy has been destroyed
2023-11-07 13:32:58,244 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.ibm.icu-64.1', not ACTIVE
2023-11-07 13:32:58,286 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.plugin.jslibs', not ACTIVE
2023-11-07 13:32:58,906 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.confluence.plugins.confluence-icons', not ACTIVE
2023-11-07 13:32:59,032 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.confluence.plugins.confluence-editor-v4', not ACTIVE
2023-11-07 13:32:59,442 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'org.apache.servicemix.bundles.spring-tx-5.1.8.RELEASE_1', not ACTIVE
2023-11-07 13:32:59,444 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'org.apache.servicemix.bundles.spring-jdbc-5.1.8.RELEASE_1', not ACTIVE
2023-11-07 13:33:00,333 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.plugins.atlassian-spring-interceptor-adapter-plugin-1.1', not ACTIVE
2023-11-07 13:33:00,359 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.plugin.atlassian-spring-scanner-runtime', not ACTIVE
2023-11-07 13:33:00,359 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.plugin.atlassian-spring-scanner-annotation', not ACTIVE
2023-11-07 13:33:00,360 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.platform.spring-velocity-support-0.0.7', not ACTIVE
2023-11-07 13:33:00,366 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.oauth.atlassian-oauth-api-4.0.2', not ACTIVE
2023-11-07 13:33:00,392 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.gadgets.atlassian-gadgets-spi-4.4.4', not ACTIVE
2023-11-07 13:33:00,393 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.gadgets.atlassian-gadgets-api-4.4.4', not ACTIVE
2023-11-07 13:33:01,777 WARN [FelixStartLevel] [plugin.osgi.factory.OsgiBundlePlugin] disableInternal Cannot disable Bundle 'com.atlassian.bundles.json-20070829.0.0.1', not ACTIVE
2023-11-07 13:33:02,017 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,027 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,050 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,069 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,071 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,072 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,180 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,212 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,490 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,492 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,513 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,515 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,530 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,532 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,547 WARN [Catalina-utility-2] [plugin.osgi.factory.OsgiBundlePlugin] uninstallInternal OSGi container not running or undefined: Will not remove bundle listener and will not close package admin service
2023-11-07 13:33:02,619 ERROR [Catalina-utility-2] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@1c33b819: service proxy has been destroyed
2023-11-07 13:33:02,620 WARN [Catalina-utility-2] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@1c33b819 took 25470 ms (warning threshold is 5000 ms)
2023-11-07 13:33:02,881 INFO [Catalina-utility-2] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2023-11-07 13:33:02,978 INFO [Catalina-utility-2] [com.atlassian.confluence.lifecycle] shutdownCacheManager Shutting down EhCache cache manager
2023-11-07 13:33:03,002 INFO [Catalina-utility-2] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence
2023-11-07 13:39:07,252 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.4.0 [build 8401 based on commit hash 0f22f74a0c673511e48ab1bc7ae6af4388aa8dbf] - synchrony version 3.1.0-master-022ca438

Then, after another startup, another, different error:

2023-11-07 13:41:01,866 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
2023-11-07 13:41:04,082 ERROR [http-nio-8090-exec-1] [opensymphony.webwork.dispatcher.VelocityResult] doExecute Unable to render Velocity Template, '/login.vm'
 -- url: /login.action | traceId: 039f910f4715a547 | userName: anonymous | action: login
org.apache.velocity.exception.MethodInvocationException: Invocation of method 'isShowSignUp' in  class com.atlassian.confluence.user.actions.LoginAction threw exception java.lang.NullPointerException: Application cannot be null at /login.vm[line 6, column 28

I also checked the database as described here: https://confluence.atlassian.com/confkb/confluence-returns-system-error-due-to-invocation-of-method-isshowsignup-in-class-com-atlassian-confluence-user-actions-loginaction-threw-exception-java-lang-nullpointerexception-at-login-vm-693899125.html

But in the database, everything seems to be in order:

confluence=# select * from cwd_application;
id | application_name | lower_application_name | created_date | updated_date | active | description | application_type | crede
ntial
--------+------------------+------------------------+-------------------------+-------------------------+--------+-------------+---------------------+------
------
262145 | crowd-embedded | crowd-embedded | 2016-02-05 08:53:35.631 | 2016-02-17 12:28:33.694 | T | | GENERIC_APPLICATION | X
(1 row)

confluence=# SELECT * FROM cwd_app_dir_mapping ORDER BY 5;
id | application_id | directory_id | allow_all | list_index
---------+----------------+--------------+-----------+------------
1409026 | 262145 | 1376258 | T | 0
1998849 | 262145 | 1966081 | T | 1
327681 | 262145 | 294913 | T | 2
(3 rows)

confluence=# select * from cwd_directory;
id | directory_name | lower_directory_name | created_date | updated_date | active | descrip
tion | impl_class | lower_impl_class | directory_type
---------+-------------------------------+-------------------------------+-------------------------+-------------------------+--------+---------------------
------------------+--------------------------------------------------------+--------------------------------------------------------+----------------
294913 | Confluence Internal Directory | confluence internal directory | 2016-02-05 08:53:35.677 | 2016-02-05 08:53:35.677 | T | Confluence default i
nternal directory | com.atlassian.crowd.directory.InternalDirectory | com.atlassian.crowd.directory.internaldirectory | INTERNAL
1966081 | Active Directory server | active directory server | 2016-02-17 12:28:23.689 | 2016-03-07 14:21:08.757 | F |
| com.atlassian.crowd.directory.MicrosoftActiveDirectory | com.atlassian.crowd.directory.microsoftactivedirectory | CONNECTOR
1376258 | XXXX | xxxx | 2016-02-08 14:06:50.183 | 2017-06-03 11:23:04.064 | T |
| com.atlassian.crowd.directory.MicrosoftActiveDirectory | com.atlassian.crowd.directory.microsoftactivedirectory | CONNECTOR

I'm not sure what else I can do and I'm not even sure where the problem is... Can anyone please help? Any ideas or proposals are welcome. This is a used system and people really rely on this instance so any assistance would be VERY much appreciated. System information below:

--------------------------
Build Information
--------------------------
maxUsers = 250
installationDate = Sun Nov 05 05:30:49 EET 2023
supportEntitlementNumber = SEN-6616071
buildNumber = 8401
serverId = BZ8C-L22U-93BT-1DEQ
version = 7.4.0
home = /var/atlassian/application-data/confluence
upTime = 2 minutes, 42 seconds
baseUrl = http://XXXXXXXXXXXXX:8090
devMode = false
--------------------------
Server Information
--------------------------
Application Server: Apache Tomcat/9.0.33
Servlet Version: 4.0
--------------------------
Database Information
--------------------------
Database Dialect: com.atlassian.confluence.impl.hibernate.dialect.PostgreSQLDialect
Database URL: jdbc:postgresql://XXXXXXXXXX:5432/confluence
Database Driver Name: org.postgresql.Driver
Database Driver Version: 42.2.8
Database Name: PostgreSQL
Database Version: 11.11
Database Latency (ms): 4
--------------------------
System Information
--------------------------
date = Tuesday, 7 November 2023
jvmInputArguments = --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/opt/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms4096m -Xmx4096m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/opt/atlassian/confluence/logs/gc-2023-11-07_13-29-46.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-11-07_13-29-46.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/confluence -Dcatalina.home=/opt/atlassian/confluence -Djava.io.tmpdir=/opt/atlassian/confluence/temp 
workingDirectory = /opt/atlassian/confluence/bin
javaVersion = 11.0.5
javaSpecificationVersion = 11
operatingSystem = Linux 3.10.0-1160.21.1.el7.x86_64
operatingSystemVersion = 3.10.0-1160.21.1.el7.x86_64
appServer = Apache Tomcat
javaVm = OpenJDK 64-Bit Server VM
javaRuntime = OpenJDK Runtime Environment
operatingSystemArchitecture = amd64
fileSystemEncoding = UTF-8
jvmInputArgumentsFiltered = -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Xms4096m -Xmx4096m -XX:+UseG1GC -XX:G1ReservePercent=20 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions
systemLanguage = en
jvmVendor = Oracle Corporation
javaVendor = AdoptOpenJDK
userName = confluence
tempDirectory = /opt/atlassian/confluence/temp
jvmImplementationVersion = 11.0.5+10
jvmVersion = 11
favouriteCharacter = Heimdall
favouriteColour = Dandelion
time = 13:32:33
systemTimezone = Europe/Xxxxxxxx
operatingSystemName = Linux
--------------------------
Global Settings
--------------------------
allowCamelCase = false
backupAttachmentsDaily = true
enableOpenSearch = true
showContactAdministratorsForm = true
draftSaveInterval = 30000
webSudoTimeout = 10
webSudoEnabled = true
dailyBackupFilePrefix = backup-
indexingLanguage = english
emailAddressVisibility = email.address.public
attachmentDataStore = file.system.based.attachments.storage
maxAttachmentsInUI = 5
gzippingResponse = true
disableLogo = false
defaultSpaceHomepageTitle = Home
maintenanceBannerMessage = null
allowRemoteApiAnonymous = false
supportRequestEmail = confluence-autosupportrequests@atlassian.com
defaultEncoding = UTF-8
addWildcardsToUserAndGroupSearches = true
defaultTimezoneId = null
attachmentSecurityLevel = SMART
showSystemInfoIn500Page = false
timeZone = time.zone.Europe.Xxxxxxxxx
saveable = true
globalDefaultLocale = en_GB
rssTimeout = 60
allowThreadedComments = true
ignoredAdminTasks = null
maxThumbDimensions = 300x300
antiXssMode = true
maxThumbHeight = 300
auditLogRetentionUnit = Years
denyPublicSignup = true
siteTitle = Confluence
siteHomePage = null
allowTrackbacks = false
xsrfAddComments = true
maxThumbWidth = 300
backupDaily = true
auditLogRetentionNumber = 3
nofollowExternalLinks = true
senMissingInLicenseMessageOff = true
almostSupportPeriodEndMessageOff = false
defaultUsersGroup = confluence-users
backupPath = /var/atlassian/application-data/confluence/backups
baseUrlAdminMessageOff = false
externalUserManagement = false
maxSimultaneousQuickNavRequests = 40
maxRssItems = 200
emailAdminMessageOff = false
loginManagerSettings = com.atlassian.confluence.setup.settings.beans.LoginManagerSettings@16aa849a
maintenanceBannerMessageOn = false
dailyBackupDateFormatPattern = yyyy_MM_dd
attachmentMaxSize = 104857600
customContactMessage = null
baseUrl = http://XXXXXXXXXXXXXXX:8090
enableSpaceStyles = false
enableQuickNav = true
allowRemoteApi = false
pageTimeout = 120
--------------------------
JVM Stats
--------------------------
xmx = 4294967296
usedNonHeap = 391911112
availableHeap = 2699647488
availableNonHeap = -1
allocatedHeap = 4294967296
freeAllocatedHeap = 2699647488
maxNonHeap = -1
availablePermGen = 0
maxPermGen = -1
maxHeap = 4294967296
usedHeap = 1595319808
usedPermGen = -1
xms = 4294967296
--------------------------
Cluster Information
--------------------------
Not clustered.

[Taaniel-Martinus Paal · Accepted solution]

We have multiple different confluence instances and each has their own database. What I failed to notice was a simple mistake (as it usually is) that in the confluence.cfg.xml, the database connection was to a different instance than the one I was checking (last IP digits different). After checking the correct instance, we noticed that the database was empty. Same happened to another confluence instance. Sadly, this was caused by a malicious party who had managed to penetrate our security barrier. Luckily - this database instance had backups ready, so we had no trouble to restore both confluence instances to their healthy state. So, to summarize - the devil is in the details and always make sure you have regular backups available :)

[Chris]

Hi, 

the only reference points I´ve found are:

[CONFSERVER-40717] Confluence slow startup because it is trying to start Hipchat plugin which have dependency on another plugin which has been disabled - Create and track feature requests for Atlassian products.

'org.osgi.framework.BundleException' problems when loading plugins due to class loading | Confluence | Atlassian Documentation

Service Proxy has Been Destroyed | Git Integration for Jira Self Managed (gitkraken.com)

(Jira Link, but it also points at a defective plugin)

How to clear Confluence plugins cache | Confluence | Atlassian Documentation

I hope this will be helpful to you in some way.

KR

Chris

[Taaniel-Martinus Paal]

Hi, Chris!

I already cleared plugin cache but that did not change anything so that's a no, sadly. I'm just wondering how it cannot find the application per the error message even though it clearly exists and has all the right mappings in the related tables... Should we try to recreate some entries manually regarding directories? Could that help? I mean removing integrations with some AD/LDAP configuration... Just at my wit's end here.

Thanks again,

Taaniel.