Confluence 4.1 backups not working

Question

I'm looking for some guidance on how to troubleshoot Confluence Backups. I turned on Diagnostic logging but there is nothing related to Scheduling or Backups in the logs. The backups stopped in May and I'm not sure what event triggered that. I've tried xml backups but I get an "out of memory" error, I'm thinking that's my best option so far (increase the memory, I've got 12g), here is my setenv: JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=1024m $JAVA_OPTS -Djava.awt.headless=true" [root@wiki confluence]# more /proc/meminfo MemTotal: 12462956 kB MemFree: 5670332 kB Buffers: 290484 kB Cached: 2487828 kB SwapCached: 0 kB Active: 4919712 kB Inactive: 1741028 kB HighTotal: 11655652 kB HighFree: 5266824 kB LowTotal: 807304 kB LowFree: 403508 kB SwapTotal: 4192956 kB SwapFree: 4192956 kB Dirty: 132 kB Writeback: 0 kB AnonPages: 3882424 kB Mapped: 73772 kB Slab: 107852 kB PageTables: 11420 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 10424432 kB Committed_AS: 6145256 kB VmallocTotal: 116728 kB VmallocUsed: 5896 kB VmallocChunk: 110608 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB [root@wiki temp]# ls xmlexport-20120818-094104-2 xmlexport-20120818-094913-3 [root@wiki temp]# ls -l total 8 drwxr-xr-x 2 root root 4096 Aug 18 09:41 xmlexport-20120818-094104-2 drwxr-xr-x 2 root root 4096 Aug 18 09:49 xmlexport-20120818-094913-3 [root@wiki temp]# cd xmlexport-20120818-094913-3/ [root@wiki xmlexport-20120818-094913-3]# ls exportDescriptor.properties [root@wiki xmlexport-20120818-094913-3]# more exportDescriptor.properties #Sat Aug 18 09:49:13 EDT 2012 exportType=all buildNumber=3126 backupAttachments=true [root@wiki xmlexport-20120818-094913-3]# df -k . Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 146764096 92742976 46445672 67% / StartedEndedDuration (ms)Aug 18, 20129:49:13 AMAug 18, 20129:50:25 AM71778 9:41:04 AM 9:42:16 AM72373 drwxr-xr-x 17 confluence1 root 4096 Aug 18 02:30 . drwxr-xr-x 3 root root 4096 Dec 16 2011 .. drwxr-xr-x 3 confluence1 confluence1 4096 Dec 16 2011 attachments drwxr-xr-x 2 confluence1 confluence1 4096 Aug 17 07:29 backups drwxr-xr-x 2 confluence1 confluence1 12288 Dec 16 2011 bundled-plugins drwxr-xr-x 2 confluence1 confluence1 4096 Dec 16 2011 config -rw-r--r-- 1 root root 2718 Aug 18 02:30 confluence.cfg.xml drwxr-xr-x 2 confluence1 confluence1 12288 Aug 18 08:45 imgEffects drwxr-xr-x 3 confluence1 confluence1 4096 Aug 18 08:46 index drwxr-xr-x 2 confluence1 confluence1 4096 Aug 17 23:58 logs drwxr-xr-x 2 root root 4096 Aug 18 02:29 plugins-cache drwxr-xr-x 5 confluence1 confluence1 4096 Dec 16 2011 plugins-osgi-cache drwxr-xr-x 2 confluence1 confluence1 53248 Aug 18 02:30 plugins-temp drwxr-xr-x 2 confluence1 confluence1 4096 Mar 8 16:53 restore drwxr-xr-x 3 confluence1 confluence1 4096 Dec 16 2011 script drwxr-xr-x 4 confluence1 confluence1 12288 Aug 18 09:49 temp drwxr-xr-x 511 confluence1 confluence1 16384 Aug 16 23:21 thumbnails drwxr-xr-x 3 confluence1 confluence1 65536 Aug 17 15:36 viewfile The Confluence service has been restared a number of times since May if that helps. Leo

Radu Dumitriu · Answer

https://confluence.atlassian.com/display/CONFKB/Cannot+Create+XML+Backup+Due+to+OutOfMemory+Errors

Some obs here:

Your MaxPermSize is anyway too high. You should set it to something like 256m / 384m, that should be enough if you do not have (too many) plugins. Now, to determine the max heap, you just need to substract from the maximal heap 2048m the maxpermsize and the memory took by the java process (usually around 256m).

For instance:

Max heap = 2048m (max process space) - 256m (native) - 384m (permgen) = 1408m maximal heap

You can run Java like:

-Xms1408m -Xmx1408m -XX:PermSize=384m -XX:MaxPermSize=384m

If you will have OOM due to perm gen space, then you need to increase the memory for perm gen, otherwise keep it like that.

Since you're on Linux - I assume -, you can boot the kernel with large memory support on 32bits, so the maximal memory you can allocate to a process will be 3Gb (process will take 4Gb native actually, but 1Gb will be used for memory mappings).

Leo Dopson · Answer

I decided to create the table, actually took more parameters than the one I have listed.

create index cl_title_idx on CONTENTLOCK (TITLE)
create index cl_content_idx on CONTENTLOCK (CONTENT)
alter table CONTENTLOCK add constraint FK692CA9046382C059 foreign key (CONTENT) references CONTENT

Now I'may be getting to the root of the issue:

I'm running on a 32bit OS, so I can not allocate 2048 of Memory, because Java refuses to start!

I have 12 gig of memory installed.

2012-08-18 14:54:59,034 ERROR [scheduler_Worker-7] [org.quartz.core.JobRunShell] run Job DEFAULT.BackupJob threw an unhandled Exception: 
java.lang.OutOfMemoryError: Java heap space
2012-08-18 14:54:59,035 ERROR [scheduler_Worker-7] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.BackupJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:210)
    at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: java.lang.OutOfMemoryError: Java heap space

Leo Dopson · Answer

Finally I see something in log file:

Should I create the CONTENTLOCK table?

create table CONTENTLOCK (CONTENTLOCKID bigint not null, TITLE varchar(20), GROUPNAME varchar(255), EXPIRYDATE timestamp, CONTENT bigint, CREATOR varchar(255), COMMENTSTR varchar(255), CREATIONDATE timestamp, LASTMODIFIER varchar(255), LASTMODDATE timestamp, primary key (CONTENTLOCKID))

2012-08-18 14:04:45,703 ERROR [scheduler_Worker-4] [confluence.importexport.impl.BackupJob] executeJob Error while running the scheduled backup
com.atlassian.confluence.importexport.ImportExportException: Couldn't backup database data.
    at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEntities(AbstractXmlExporter.java:197)
    at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.backupEverything(AbstractXmlExporter.java:99)
    at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupEverything(FileXmlExporter.java:83)
    at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.doExport(AbstractXmlExporter.java:92)
    at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExport(FileXmlExporter.java:42)

...

at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.springframework.jdbc.BadSqlGrammarException: Hibernate operation: Could not execute query; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluence.CONTENTLOCK' doesn't exist