Confluence restore APPARENT DEADLOCK

NWS AI June 3, 2019

I'm trying to migrate Confluence from H2 to a new DB engine (PostgreSQL) and when installing the new instance I need to restore from the backup (540MB). It is because I accidentally installed it using H2 in the first place even if it was for production purposes. 

The problem is that while trying to restore the backup it gets stuck to 12%, gives time estimates between 7 and 11 hours but it doesn't make any progress.

 

The atlassian-confluence.log says APPARENT DEADLOCK (attached the stack trace).

 

What is the best way of restoring a +500MB backup and what should be the settings?

Before getting the APPARENT DEADLOCK error I was previously getting a 

PSQLException: This connection has been closed.

while trying to restore to an AWS RDS PostgreSQL.

 

/setup/setupdbtype.action | traceId: 00e4f8a94643d567 | userName: anonymous | action: setupdbtype
2019-06-03 15:24:08,768 WARN [C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@373830f7 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
-- referer: /setup/setupdbtype.action | url: /setup/setupdbtype.action | traceId: 00e4f8a94643d567 | userName: anonymous | action: setupdbtype
2019-06-03 15:24:42,671 WARN [C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-AdminTaskTimer] [mchange.v2.async.ThreadPoolAsynchronousRunner] log com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@373830f7 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@fe679fd
on thread: C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3f9b0fce
on thread: C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#2
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@4ae4e665
on thread: C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#1
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7b64af7f
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@47f1187f
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5637839c
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@43445e1d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7bd75d2
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@37ed5d64
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@48aa210
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1963d20f
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#2,5,main]
java.lang.Class.getDeclaredMethods0(Native Method)
java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
java.lang.Class.privateGetMethodRecursive(Class.java:3048)
java.lang.Class.getMethod0(Class.java:3018)
java.lang.Class.getMethod(Class.java:1784)
com.mchange.v2.c3p0.impl.C3P0ImplUtils.supportsMethod(C3P0ImplUtils.java:286)
com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:117)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:240)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:722)
Thread[C3P0PooledConnectionPoolManager[identityToken->1br9rnha389qlusc2cebo|69c07976]-HelperThread-#0,5,main]
java.util.HashSet.<init>(HashSet.java:106)
com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:81)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:240)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

 

1 answer

0 votes
Stephen Sifers
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 7, 2019

Hello Bogdan,

Thank you for providing the steps you took prior to this error and the output of the error message you’re receiving. The error within your reported logs is an almost mirrored match to a KB we have which discusses this issue and shows ways to resolve it. You may find this article at Troubleshooting deadlocks in Confluence. The causes and resolutions to this issue are as follows:

Causes and Resolutions

Deadlocks can result from a variety of different causes:

  1. Database connection pool has too few connections. See Confluence slows and times out during periods of high load due to database connection pool
  2. An Confluence crashes due to 'OutOfMemoryError Java heap space' error is occurring. Check that page for diagnosis and resolutions.
  3. A script or Remote API process is hammering the server. Disable any Remote API scripts or disable the Remote API from Administration >> General Information.
  4. The database is not optimized.
  5. If you are using MS SQL Server, check if you are affected by this known deadlock problem.
  6. Adding pages concurrently can result in deadlocks. See CONFSERVER-14789.

If you’re still having issues after going through the above options, please reach back out and let us know.

Regards,
Stephen Sifers

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events