nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Idle Confluence instance causes constant write load on disk/SSD

Patrick_M__Hausen

Hi all,

I run a small self hosted Confluence 7.5.1 instance. 10 user license, not much load.

My system monitoring sees a constant write load of about 600k/s to disk with just Confluence and MySQL running and definitely no active users.

What can I check to find the cause of this?

Operating system is Ubuntu 20.04 server, no graphical user interface, just this one application.

Thanks,
Patrick

Bildschirmfoto 2020-06-12 um 12.21.31.png

Accepted answers

All comments

Amith Mathur {Appfire}

Hi @Patrick_M__Hausen ,

Welcome to Atlassian Community!

There will be many reasons for these types of performance related issues. Is the instance accessible at those times? Is it happening at any particular time? You can go through the below documents to get to the root cause for it.

Ensure that the backups and any scheduler/scripts are disabled.

Thanks,
Amith Mathur

Patrick_M__Hausen

The instance is perfectly accessible and performing well. This is not a performance issue. It is just that this server is writing to the SSD 24x7 which will possibly wear it out sooner than desired.

An idle application should not continuously write apart from some bookkeeping every couple of minutes or so, shouldn't it?

I'll try the suggested diagnostics. Thanks.

repi

Which apps are installed?
Are there any settings regarding the configuration of the log file that cause the confluence to write detailed logs?
Is the log level production or diagnostic?

Patrick_M__Hausen

Confluence Source Editor
Polls for Confluence
Team Calendars

2. not that i knew

3. production

Patrick_M__Hausen

OK, iotop tells me it's the database (mysqld) that is writing, so I enabled the SQL query log for a short period of time. I get a bunch of these:

2020-06-12 16:02:14,815 ERROR [Caesium-1-2] [migration.agent.queue.QueueBroker] error An error occurred when getting the next batch for consumer type: CONFLUENCE_IMPORT. Message: javax.persistence.PersistenceException: Failed to update database schema

com.atlassian.util.concurrent.LazyReference$InitializationException: javax.persistence.PersistenceException: Failed to update database schema

at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)

at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)

at com.atlassian.migration.agent.store.jpa.impl.DefaultSessionFactorySupplier.get(DefaultSessionFactorySupplier.java:65)

at com.atlassian.migration.agent.store.jpa.impl.DefaultSessionFactorySupplier.get(DefaultSessionFactorySupplier.java:40)

at com.atlassian.migration.agent.store.jpa.impl.DefaultPluginTransactionTemplate.on(DefaultPluginTransactionTemplate.java:29)

at com.atlassian.migration.agent.store.tx.PluginTransactionTemplate.write(PluginTransactionTemplate.java:24)

at com.atlassian.migration.agent.queue.QueueBroker.getNextBatch(QueueBroker.java:119)

at com.atlassian.migration.agent.queue.QueueBroker.dispatchBatchToConsumer(QueueBroker.java:113)

at java.base/java.util.ArrayList.forEach(Unknown Source)

at com.atlassian.migration.agent.queue.QueueBroker.runJob(QueueBroker.java:100)

at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:117)

at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.lambda$runJob$0(JobRunnerWrapper.java:87)

at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84)

at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68)

at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:87)

at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)

at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)

at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:402)

at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:380)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)

at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: javax.persistence.PersistenceException: Failed to update database schema

at com.atlassian.migration.agent.store.jpa.impl.LiquibaseSchemaUpgrader.upgrade(LiquibaseSchemaUpgrader.java:35)

at com.atlassian.migration.agent.store.jpa.impl.DefaultSessionFactorySupplier.buildSessionFactory(DefaultSessionFactorySupplier.java:70)

at com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)

at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)

at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)

at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)

at com.atlassian.migration.agent.store.jpa.impl.DefaultSessionFactorySupplier.get(DefaultSessionFactorySupplier.java:65)

at com.atlassian.migration.agent.store.jpa.impl.DefaultSessionFactorySupplier.get(DefaultSessionFactorySupplier.java:40)

at com.atlassian.migration.agent.store.jpa.impl.DefaultPluginTransactionTemplate.on(DefaultPluginTransactionTemplate.java:29)

at com.atlassian.migration.agent.store.tx.PluginTransactionTemplate.read(PluginTransactionTemplate.java:16)

at com.atlassian.migration.agent.service.impl.DefaultStatisticsService.loadSpaceStatistics(DefaultStatisticsService.java:138)

at com.atlassian.migration.agent.service.impl.DefaultStatisticsService.calculateServerStats(DefaultStatisticsService.java:167)

at com.atlassian.migration.agent.service.impl.SingleJobExecutor.lambda$execute$0(SingleJobExecutor.java:33)

at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

... 1 more

Caused by: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by 2003:a:d59:3800:2a0:98ff:fe4f:2112%enp0s4 (2003:a:d59:3800:2a0:98ff:fe4f:2112%enp0s4) since 11/6/19 12:12 PM

at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:230)

at liquibase.Liquibase.update(Liquibase.java:184)

at liquibase.Liquibase.update(Liquibase.java:179)

at com.atlassian.migration.agent.store.jpa.impl.LiquibaseSchemaUpgrader.upgrade(LiquibaseSchemaUpgrader.java:30)

... 16 more