Confluence 4.1 backups not working

[Leo Dopson]

I'm looking for some guidance on how to troubleshoot Confluence Backups.

I turned on Diagnostic logging but there is nothing related to Scheduling or Backups in the logs. The backups stopped in May and I'm not sure what event triggered that. I've tried xml backups but I get an "out of memory" error, I'm thinking that's my best option so far (increase the memory, I've got 12g), here is my setenv: JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=1024m $JAVA_OPTS -Djava.awt.headless=true"

[root@wiki confluence]# more /proc/meminfo
MemTotal: 12462956 kB
MemFree: 5670332 kB
Buffers: 290484 kB
Cached: 2487828 kB
SwapCached: 0 kB
Active: 4919712 kB
Inactive: 1741028 kB
HighTotal: 11655652 kB
HighFree: 5266824 kB
LowTotal: 807304 kB
LowFree: 403508 kB
SwapTotal: 4192956 kB
SwapFree: 4192956 kB
Dirty: 132 kB
Writeback: 0 kB
AnonPages: 3882424 kB
Mapped: 73772 kB
Slab: 107852 kB
PageTables: 11420 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 10424432 kB
Committed_AS: 6145256 kB
VmallocTotal: 116728 kB
VmallocUsed: 5896 kB
VmallocChunk: 110608 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

[root@wiki temp]# ls
xmlexport-20120818-094104-2 xmlexport-20120818-094913-3
[root@wiki temp]# ls -l
total 8
drwxr-xr-x 2 root root 4096 Aug 18 09:41 xmlexport-20120818-094104-2
drwxr-xr-x 2 root root 4096 Aug 18 09:49 xmlexport-20120818-094913-3
[root@wiki temp]# cd xmlexport-20120818-094913-3/
[root@wiki xmlexport-20120818-094913-3]# ls
exportDescriptor.properties
[root@wiki xmlexport-20120818-094913-3]# more exportDescriptor.properties
#Sat Aug 18 09:49:13 EDT 2012
exportType=all
buildNumber=3126
backupAttachments=true

[root@wiki xmlexport-20120818-094913-3]# df -k .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 146764096 92742976 46445672 67% /

<th colspan="2">Started</th><th colspan="2">Ended</th><th>Duration (ms)</th>

Aug 18, 2012	9:49:13 AM	Aug 18, 2012	9:50:25 AM	71778
	9:41:04 AM		9:42:16 AM	72373

drwxr-xr-x 17 confluence1 root 4096 Aug 18 02:30 .
drwxr-xr-x 3 root root 4096 Dec 16 2011 ..
drwxr-xr-x 3 confluence1 confluence1 4096 Dec 16 2011 attachments
drwxr-xr-x 2 confluence1 confluence1 4096 Aug 17 07:29 backups
drwxr-xr-x 2 confluence1 confluence1 12288 Dec 16 2011 bundled-plugins
drwxr-xr-x 2 confluence1 confluence1 4096 Dec 16 2011 config
-rw-r--r-- 1 root root 2718 Aug 18 02:30 confluence.cfg.xml
drwxr-xr-x 2 confluence1 confluence1 12288 Aug 18 08:45 imgEffects
drwxr-xr-x 3 confluence1 confluence1 4096 Aug 18 08:46 index
drwxr-xr-x 2 confluence1 confluence1 4096 Aug 17 23:58 logs
drwxr-xr-x 2 root root 4096 Aug 18 02:29 plugins-cache
drwxr-xr-x 5 confluence1 confluence1 4096 Dec 16 2011 plugins-osgi-cache
drwxr-xr-x 2 confluence1 confluence1 53248 Aug 18 02:30 plugins-temp
drwxr-xr-x 2 confluence1 confluence1 4096 Mar 8 16:53 restore
drwxr-xr-x 3 confluence1 confluence1 4096 Dec 16 2011 script
drwxr-xr-x 4 confluence1 confluence1 12288 Aug 18 09:49 temp
drwxr-xr-x 511 confluence1 confluence1 16384 Aug 16 23:21 thumbnails
drwxr-xr-x 3 confluence1 confluence1 65536 Aug 17 15:36 viewfile

The Confluence service has been restared a number of times since May if that helps.

Leo

[Radu Dumitriu]

https://confluence.atlassian.com/display/CONFKB/Cannot+Create+XML+Backup+Due+to+OutOfMemory+Errors

Some obs here:

Your MaxPermSize is anyway too high. You should set it to something like 256m / 384m, that should be enough if you do not have (too many) plugins. Now, to determine the max heap, you just need to substract from the maximal heap 2048m the maxpermsize and the memory took by the java process (usually around 256m).

For instance:

Max heap = 2048m (max process space) - 256m (native) - 384m (permgen) = 1408m maximal heap

You can run Java like:

-Xms1408m -Xmx1408m -XX:PermSize=384m -XX:MaxPermSize=384m

If you will have OOM due to perm gen space, then you need to increase the memory for perm gen, otherwise keep it like that.

Since you're on Linux - I assume -, you can boot the kernel with large memory support on 32bits, so the maximal memory you can allocate to a process will be 3Gb (process will take 4Gb native actually, but 1Gb will be used for memory mappings).

[Leo Dopson]

I decided to create the table, actually took more parameters than the one I have listed.

create index cl_title_idx on CONTENTLOCK (TITLE)
create index cl_content_idx on CONTENTLOCK (CONTENT)
alter table CONTENTLOCK add constraint FK692CA9046382C059 foreign key (CONTENT) references CONTENT

Now I'may be getting to the root of the issue:

I'm running on a 32bit OS, so I can not allocate 2048 of Memory, because Java refuses to start!

I have 12 gig of memory installed.

2012-08-18 14:54:59,034 ERROR [scheduler_Worker-7] [org.quartz.core.JobRunShell] run Job DEFAULT.BackupJob threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
2012-08-18 14:54:59,035 ERROR [scheduler_Worker-7] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.BackupJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:210)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: java.lang.OutOfMemoryError: Java heap space

[Leo Dopson]

Finally I see something in log file:

Should I create the CONTENTLOCK table?

create table CONTENTLOCK (CONTENTLOCKID bigint not null, TITLE varchar(20), GROUPNAME varchar(255), EXPIRYDATE timestamp, CONTENT bigint, CREATOR varchar(255), COMMENTSTR varchar(255), CREATIONDATE timestamp, LASTMODIFIER varchar(255), LASTMODDATE timestamp, primary key (CONTENTLOCKID))

2012-08-18 14:04:45,703 ERROR [scheduler_Worker-4] [confluence.importexport.impl.BackupJob] executeJob Error while running the scheduled backup
com.atlassian.confluence.importexport.ImportExportException: Couldn't backup database data.
at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEntities(AbstractXmlExporter.java:197)
at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEverything(AbstractXmlExporter.java:99)
at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupEverything(FileXmlExporter.java:83)
at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.doExport(AbstractXmlExporter.java:92)
at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExport(FileXmlExporter.java:42)

...

at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.springframework.jdbc.BadSqlGrammarException: Hibernate operation: Could not execute query; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluence.CONTENTLOCK' doesn't exist