Can't seem to export my MindTouch wiki with UWC

[James McCarthy]

Hello everyone,

We are trying to export our old MindTouch (Dekiwiki) information so that we can import in to confluence, but we are having some issues. I will include my config and error information below:

When I don't include the :8443 in the url.base, it doesn't connect at all. It gives me a 404 error. When I set my config to be below it gets past the 404 and then gives an error of

"2011-10-05 11:12:42,423 INFO [Thread-4] - Exporting Mindtouch...
2011-10-05 11:12:42,424 INFO [Thread-4] - Getting page tree.
2011-10-05 11:12:43,611 INFO [Thread-4] - basic authentication scheme selected
2011-10-05 11:12:43,612 INFO [Thread-4] - No credentials available for BASIC 'DekiWiki'@wiki.myurl.com:8443
2011-10-05 11:12:43,612 ERROR [Thread-4] - Problem encountered when making rest call: 401: HTTP/1.1 401 Unauthorized
2011-10-05 11:12:43,612 ERROR [Thread-4] - Problem occured when making rest request to: https://wiki.myurl.com:8443/@api/deki/pages/?authenticate=true"

With the below config I get the above error:

url.base=https://wiki.myurl.com:8443
## Port used by your Mindtouch. Default is 8443.
url.port=8443

I have verified that the link above in my url.base returns a login prompt in the browser. When I fill in the login credentials, it connects and gives me the correct xml page with a list of all of our wiki pages.

Can anyone suggest anything here? We are running with ldap disabled and only using local authentication for the wiki and I have verified the login credentials are correct in the config.

I'm not sure how to tell what version of REST api we are using but MindTouch is version 9.12.3.

Thanks in advance for any help!!

[Laura Kolker]

It goes through every page, thousands, but doesn't create anything locally. I'm still working on it but we shal see..

The log is flooded with these entries and it goes through all of our pages and ends with an error. Nothing gets created in the directory I specify. I made sure that the directory has full read and write access.

Have you tried grepping the log for 404s? If there are missing attachments in your mindtouch that still have links to them, sometimes this triggers a fatal problem. It's not something I was ever able to reproduce, and my attempts to fix it without a platform on which to debug didn't end up working. Once in a while, someone will tell me they arranged a fix, but unfortunately, no patches submitted to date. So, if you see 404 errors in the log, use the accompanying info to look for the bad links so you can clean up the mindtouch data. It can be a bit of a chore, but upside: your data's gotten a scrubbing before migration.

If you don't see 404s, then unfortunately, I'm out of ideas.

[James McCarthy]

Yeah, I don't see any 404's anywhere. Though at the end of the log now I'm getting a "can't create directory" type of error. This is after 10 minutes of it running through each page. It's not getting to the point where its creating the mindtouch export directory inside of the folder I tell it to. It's like its going through the process but skipping the part where it copies the data to a new directory. I'm assuming AppFusions would offer a service to help with our issue right now but it would cost a significant amount of money, yes? :0)

Please advise if you can, if not, totally understandable.

Thanks!

[James McCarthy]

Well, I think I got it to the point where it now exports all pages until the first one with attachments. Then it quits. So I'm thinking about just downloading all of the attachments manually and removing them from the pages? Not sure if that's a valid train of thought or not... At this point, I think I'm just thinking out loud.

Thanks again!

[James McCarthy]

As a side note and for future reference, it seems that I was able to get past most of the issues. It looks like the program was having an issue with any attachments OR pages that had been restored inside of MindTouch. My solution was to go back in and delete those attachments or pages and run the script again. It's actually really easy to do that with a large site by just searching for: (restored) in your MindTouch wiki. This brings up every page with an attachment that has been restored. I literally just had to delete those and now I'm getting all normal attachments and web pages exported perfectly.

I haven't started the conversion / upload process yet to Confluence, so who knows if I am near completion or not, but we will see.

[Laura Kolker]

Hrm. This message:

2011-10-05 11:12:43,612 INFO [Thread-4] - No credentials available for BASIC 'DekiWiki'@wiki.myurl.com:8443

makes me think that the Http library is complaining that it is expecting to connect with Basic Authentication, but it can't on your network. Things you can do:

• Turn on debug messages. This might provide additional debug details.
• Pursue a networking fix. (Maybe your networking team can tune the auth configuration to keep apache from complaining)
• Download/Build against the UWC source, and alter the http connection code in the MindtouchExporter to match your authentication layer, in the event that a networking solution is inappropriate/inconvenient/etc.

[James McCarthy]

I think the issue we are having is that the error you referenced above makes it seem like it's trying to pull up a BASIC login at the url "wiki.myurl.com:8443". That won't work because that url has a web based login page. Now, if it tried to use BASIC against the url in the other part of the message, it would work. When I pull up https://wiki.myurl.com:8443/@api/deki/pages/?authenticate=true it gives me a BASIC authentication prompt as it should. It seems like the script is using the wrong url to get to the xml page and I can't figure out how to change that.

Any thoughts?

[Laura Kolker]

Some thoughts:

• With our sample mindtouch, we didn't do any clever networking, basic auth or otherwise. So, we logged in through the website, like you're describing, but we didn't see this problem. So, I'm not 100% sure that changing where the authentication layer logs in would fix the problem.
• But either way, there aren't any additional configuration options to explore really, so I still think the path forward has to be either a networking change or a source change.

[James McCarthy]

Well, the problem I have is that we want to move to this application (Confluence) but we have a very large amount of work invested in MindTouch. Without the ability to export and then import the data, purchasing our 25 user license doesn't seem feasible. I know this isn't your problem at all, but my hope is that someone at Atlassian has a way of getting us that service. We have a very vanilla / plain MindTouch install and a completely open network internally, so I can't imagine why the UWC isn't working at all. It seems to be an issue with what the script wants to do with authentication vs. what MindTouch is asking for. Without being an expert on either application, my hope was that the experts on UWC would be able to help.

At this point, maybe you can point me in the direction of someone that can assist further? Asking people to modify source code is definitely not a valid support option. (Though I do completely understand that you yourself may be limited in what answers you can provide) Do I need to open a support case? Could they even help with what we are trying to do? Or is it the case that this UWC tool isn't officially support by Atlassian?

Thanks for the help so far and I hope we can get this resolved.

[Laura Kolker]

Well, so the UWC is a free 3rd party application. No, Atlassian doesn't officially support it. Yes, I can answer some questions, although the reality is that environment problems are problematic to debug remotely. Confluence migrations – like all migrations – are highly dependent on content, context, syntax, environments, from-system parameters, and even human-input – and so providing a blanket solution answer is difficult.

The UWC is a tool to help you along the way – not a solution by itself. We suggest this checklist as a first review of your systems – and how to plan and progress forward on what development might be required to best automate this effort and mitigate risk.

Regarding further assistance, you could negotiate with an Atlassian partner to arrange for custom support. AppFusions happens to be available to provide that sort of contract based support, if that's what you're looking for. UWC updates to support Confluence 4 are in active review as I write. Please contact migrations@appfusions.com for more details.

[James McCarthy]

Hey, thank you for the update. I also have one of my own.

I reconfigured MindTouch to be without SSL and back on port 80 and now the UWC connects perfectly when I hit export. The problem I'm getting now is that it says successful on everything and creates the folder for me, but there is nothing in it. No pages get exported. Has this happened before as well?

[David at David Simpson Apps]

Without direct access to the system, it's always difficult trying to debug by proxy.

People like @Laura Kolker do the best that they can with the information that they have, but it's tricky. Installation X is likely to be very different to installation Y.

[Laura Kolker]

The problem I'm getting now is that it says successful on everything and creates the folder for me, but there is nothing in it. No pages get exported. Has this happened before as well?

Turn on debug messages, and keep an eye out for 404s caused by missing attachments. That's a common cause of that kind of problem.

[James McCarthy]

Here is the log file with debugging turned on:

2011-10-06 13:37:30,488 DEBUG [AWT-EventQueue-0] - Saving Setting 'WIKITYPE' = 'mindtouch'
2011-10-06 13:37:33,101 DEBUG [AWT-EventQueue-0] - Saving Current Tab = 1
2011-10-06 13:37:35,746 DEBUG [AWT-EventQueue-0] - feedback = OK
2011-10-06 13:37:35,747 DEBUG [AWT-EventQueue-0] - permissions = The user was authenticated on the server
and has the following permissions on the space with
space key:'WIKI'[view, modify, comment, admin]
2011-10-06 13:37:38,217 DEBUG [AWT-EventQueue-0] - Saving Current Tab = 0
2011-10-06 13:37:49,356 DEBUG [Thread-2] - Shutting Down...
2011-10-06 13:37:49,358 DEBUG [Thread-2] - Saving All Settings
2011-10-06 14:35:54,537 DEBUG [AWT-EventQueue-0] - Saving Setting 'WIKITYPE' = 'mindtouch'
2011-10-06 14:35:56,167 INFO [Thread-4] - Exporting Mindtouch...
2011-10-06 14:35:56,167 INFO [Thread-4] - Getting page tree.
2011-10-06 14:35:56,364 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 14:35:56,695 DEBUG [Thread-4] - Parsing page list xml.
2011-10-06 14:35:56,947 INFO [Thread-4] - Getting page content.
2011-10-06 14:35:56,949 INFO [Thread-4] - Getting page tags.
2011-10-06 14:35:56,950 INFO [Thread-4] - Getting page comments.
2011-10-06 14:35:56,950 INFO [Thread-4] - Getting attachment info.
2011-10-06 14:35:56,951 INFO [Thread-4] - Creating output directory: C:\Export\exported_mindtouch_pages
2011-10-06 14:35:56,951 INFO [Thread-4] - Writing to file system.
2011-10-06 14:35:56,951 INFO [Thread-4] - Export Complete.
2011-10-06 14:36:46,808 DEBUG [AWT-EventQueue-0] - Saving Setting 'PAGES' = ''
2011-10-06 14:36:56,210 DEBUG [Thread-2] - Shutting Down...
2011-10-06 14:36:56,211 DEBUG [Thread-2] - Saving All Settings

[James McCarthy]

I understand that. I really wasn't coming down on her at all. I was just getting worried that we would be left to our own devices for figuring out how to get our content in to confluence as rewriting the code just really wasn't an option. We are making progress now and having read some of her other posts, I know she is VERY good at this stuff so I'm confident we will figure it out!

[James McCarthy]

I may have figured it out actually. I set the ignore Mindtouch field to false and now it's pulling all content. I'll update later if I have an issue, but at the moment I think we are in the clear! Thank you again for everything so far...

[James McCarthy]

Spoke too soon... It goes through every page, thousands, but doesn't create anything locally. I'm still working on it but we shal see...

[James McCarthy]

Here is the log output:

2011-10-06 15:13:59,417 DEBUG [Thread-4] - ...Getting content for: 246_Star2005R2ClientInstall.xml
2011-10-06 15:13:59,440 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,498 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,579 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,646 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,679 DEBUG [Thread-4] - ...Getting content for: 101_WBTManager.xml
2011-10-06 15:13:59,707 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,753 DEBUG [Thread-4] - ...Getting content for: 102_EnvironmentalVariables.xml
2011-10-06 15:13:59,779 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,850 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:13:59,882 DEBUG [Thread-4] - ...Getting content for: 200_Cases.xml
2011-10-06 15:13:59,905 INFO [Thread-4] - basic authentication scheme selected
2011-10-06 15:14:00,260 DEBUG [Thread-2] - Shutting Down...
2011-10-06 15:14:00,261 DEBUG [Thread-2] - Saving All Settings

The log is flooded with these entries and it goes through all of our pages and ends with an error. Nothing gets created in the directory I specify. I made sure that the directory has full read and write access.

[Ellen Feaheny [AppFusions]]

I'm assuming AppFusions would offer a service to help with our issue right now but it would cost a significant amount of money, yes? :0)

James -

"Significant" depends on how you value solutions for your problems. It is engineering consulting with seasoned expertise to get you to your goal directly - not just throwing darts.

I think Laura stated some good advice very clearly in a few ways in the thread, as well as ideas on how to get serviced properly - and you have ignored this it seems to me. Almost to an offense.

AppFusions or any Atlassian partner does not charge more than what the cost of a problem/service costs (== time). (And yes, we have done this variation of a migration.)

No, we do not work for free.

Do you work for free? Why should engineering services be free?

Best,

Ellen

[James McCarthy]

I think you just took this conversation to a different level and have more or less ensured that we will not be clients of Confluence in the process. I will be contact Atlassian directly at this point so they can review these email threads. At no point was I offensive. At no point was I asking rhetorical, instigative questions ("Do you work for free? Why should engineering services be free?") and at no point did I say I was not willing to pay whatever price was given to me for these services. We are a client that will be in the greater than 25 user license range for Atlassian which represents a fair amount of money. I understand you are a third party company, but with your last post, you would have been better off keeping your thoughts to yourself. At this point, you should sincerely hope that Atlassian can still convince me to use their product. You have ensured we will not be utilizing your company for any services what so ever. No company should EVER speak in the manner you just have to a potential client. It is a fairly simple concept.

James

[Ellen Feaheny [AppFusions]]

Have you tried grepping the log for 404s? If there are missing attachments in your mindtouch that still have links to them, sometimes this triggers a fatal problem. It's not something I was ever able to reproduce, and my attempts to fix it without a platform on which to debug didn't end up working. Once in a while, someone will tell me they arranged a fix, but unfortunately, no patches submitted to date. So, if you see 404 errors in the log, use the accompanying info to look for the bad links so you can clean up the mindtouch data. It can be a bit of a chore, but upside: your data's gotten a scrubbing before migration.

If you don't see 404s, then unfortunately, I'm out of ideas.

[James McCarthy]

# Exporter class - class that controls the export. Don't change this.
exporter.class=com.atlassian.uwc.exporters.MindtouchExporter
###### SET THESE ######
## Base Url of your Mindtouch
## could be: 192.168.1.100, http://localhost, https://, etc
url.base=https://wiki.myurl.com:8443
## Port used by your Mindtouch. Default is 8443.
url.port=8443
## Login Credentials for your Mindtouch. Should be an admin.
user=admin
pass=admin
## Output Dir - where the exported files are put
output.dir=C:\Export
###### OPTIONAL ######
## user agent for the Xml Parser. Default is Universal Wiki Converter
user-agent=Universal Wiki Converter
## buffer size in bytes for downloading attachments. Default is 1024
att.buffer=1024
## connection timeout in ms. Default is 10000
timeout=10000
## if true, ignore page with name MindTouch. Default is true
ignore.mindtouch=true

[Joe Clark]

Would you mind posting your full exporter.mindtouch.properties configuration here (sans password)?