Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Export spaces vua command line(using SOAP remote API) times out

Maya Srinivasan November 2, 2011

Hi !

I am working with a standalone Confluence installations (3.5.6) and I'd like to export spaces from the commandline (Confluence CLI seems to be the answer).

When I try to export a reasonably large wiki space using Confluence CLI (which uses SOAP Remote API), I get:

Remote error: ; nested exception is:
Java.net.SocketTimeoutException: Read timed out

It times out in 10mins. Exactly.

Export space does work from within Confluence, and takes about 22 minutes. I've tested the export with a smaller space, and it works just fine through the CLI.

What is the right way to enable an end user to change the default timeout for all TCP connections or those initiated in a JVM so that these longer running tasks can finish and return? And, more importantly, is this the way to do it?

Has anyone had success exporting large spaces via command line?

Thanks.Maya

6 answers

1 accepted

3 votes
Answer accepted
Adam Laskowski
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 28, 2011

Maya,

I don't quite know how to get around this issue with the CLI tool for Confluence, but I was able to overcome this issue with a Python script that might be useful:

#!/usr/bin/python

import sys, string, xmlrpclib, re, httplib, time

#http timeout override
class TimeoutHTTPConnection( httplib.HTTPConnection ):
   def connect( self ):
       httplib.HTTPConnection.connect( self )
       self.sock.settimeout( self.timeout )

class TimeoutHTTP( httplib.HTTP ):
   _connection_class = TimeoutHTTPConnection
   def set_timeout( self, timeout ):
       self._conn.timeout = timeout

class TimeoutTransport( xmlrpclib.Transport ):
    def __init__( self, timeout = 10, *l, **kw ):
        xmlrpclib.Transport.__init__( self, *l, **kw )
        self.timeout = timeout
    def make_connection( self, host ):
        conn = TimeoutHTTP( host )
        conn.set_timeout( self.timeout )
        return conn

class TimeoutServerProxy( xmlrpclib.ServerProxy ):
    def __init__( self, uri, timeout = 10, *l, **kw ):
        kw['transport'] = TimeoutTransport( timeout = timeout, use_datetime = kw.get( 'use_datetime', 0 ) )
        xmlrpclib.ServerProxy.__init__( self, uri, *l, **kw )


def runExport(newtimeout):
	#server = xmlrpclib.ServerProxy('http://localhost:8080/rpc/xmlrpc')

	try:
		print ("Starting at: "+time.asctime())
		server = TimeoutServerProxy( 'http://localhost:8080/rpc/xmlrpc', timeout = newtimeout )
		token = server.confluence1.login( 'admin', 'admin' )
		key = raw_input( "Enter space key: " )
		vartype = raw_input( "Eport type (xml or html): " )
		if "xml" in vartype.lower():
			type = 'TYPE_XML'
		else: 
			if "html" in vartype.lower():
				type = 'TYPE_HTML'
			else: print ("Invalid output type.")
		string = server.confluence1.exportSpace( token, key, type )
		exit( string )
	except Exception as e:
		print (time.asctime())
		print (e)
	finally:
		print (time.asctime())
	
toset = int(raw_input( "Set Timeout Value in Minutes: " ))*60
runExport(toset)

You will need to change the details in 'server' and 'token' to match your server URL and login details. When you run the script, set a timeout value in minutes and specify the space key and xml and it should eventually spit out a URL. In local testing, a space with 35000 unique pages, each with one image attached, finished in 16 minutes. Mileage will vary, but you can get a feel for how long you can expect it to take by running a space export in the Confluence UI to establish your baseline.

0 votes
Sam Kenny
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 15, 2014

To get this to work for me, I commented out:

server = TimeoutServerProxy( 'http://localhost:8080/rpc/xmlrpc', timeout = newtimeout )

and uncommented

#server = xmlrpclib.ServerProxy('http://localhost:8080/rpc/xmlrpc')

Because my Space was quite large (3gigs) I also had to increase the amount of memory available. In my setenv.bat file I updated the JAVA_OPTS to be:

set JAVA_OPTS=%JAVA_OPTS% -Xms2056m -Xmx3072m -XX:MaxPermSize=3072m

0 votes
Bob Swift OSS (Bob Swift Atlassian Apps)
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 10, 2013

More recent versions of the Confluence CLI have a connectionTimeout parameter - Documentation

0 votes
Maya Srinivasan January 2, 2012

Adam - What version of Python are you using? I installed 2.7 (on solaris) and it gave me this error - Set Timeout Value in Minutes: 40
Starting at: Tue Jan 3 11:36:19 2012
Tue Jan 3 11:36:19 2012
TimeoutHTTP instance has no attribute 'getresponse'
Tue Jan 3 11:36:19 2012

Thanks!

Maya Srinivasan January 2, 2012

Here is the link I found about a related problem - http://stackoverflow.com/questions/5663946/problem-with-xmlrpc-server

0 votes
Maya Srinivasan November 17, 2011

I've sent the same query to the Java folks who happen to sit on my floor - I hope to hear back. It would be so very nice if Confluence would just add a way for a space admin to backup their space on a schedule.

More as I learn it.

0 votes
Joe Clark
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 2, 2011

I was just about to link you to https://studio.plugins.atlassian.com/browse/CSOAP-74 but I see that you have already commented on that, and it lead you here. :-)

You say you have tried setting CONNECTION_TIMEOUT to 1000 and it times out at around 10 minutes. If your export directly from Confluence takes around 22 minutes, you will probably need to change the timeout value to something even larger.


Try setting CONNECTION_TIMEOUT and SO_TIMEOUT to 1800000, which should give Confluence a 30 minute window to complete the export.

Maya Srinivasan November 2, 2011

The peculiar thing seems to be that no matter what I specify on the command line for the CONNECTION_TIMEOUT and SO_TIMEOUT, it still bombs out in 10mins. So, somewhere the 10min timeout for something is taking precedence....I'm hoping someone who understands the SOAP API/Java interaction better or who better know how the long running tasks in confluence work can shed some light on this.

From the description in CSOAP-74, this problem has been around a while....

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events