Don't Let Your Users End up in a Dead End!

[#1]

(Image Credit: https://www.volacci.com/blog/drupal-module-redirect)

[Big thanks to my old colleague @Ed Bukoski from Netflix, who did pioneering work in developing a redirector they used for their successful migration to Confluence Cloud.]

Picture it: you're almost at the finish line. After weeks of planning, testing, and more re-testing, you've figured out how to migrate your on-prem Confluence to the Cloud. All of the pesky apps are set to migrate properly, and you are ready.

But then your VP asks, "Hey Darryl, what's going to happen to old Confluence page links in emails or Word docs? Those are still going to work, right?" 

You break into a cold sweat, thinking about those potentially broken links. They're everywhere: emails, docs, spreadsheets, presentations. And oh, what about integrated tools like Testrail, Salesforce, Zendesk? Internal apps? Oh no….

"Darryl? Hey Darryl, are you ok? You zoned out for a minute."

You reply, "Ah yeah, no, those will keep working. We're going to put in a redirector."

What's a redirector?

A redirector is when you configure your old site (or spin up a much smaller instance) to have something like NGINX or Apache accept requests for the old URL, and then redirect them to the new URL.

When you migrate from on-prem to Confluence Cloud, the URL of your site will change from something like

https://confluence.YOURCOMPANY.com to 

https://YOURCOMPANY.atlassian.net/wiki

UPDATE: Per MIG-507 , you can now get a mapping file directly from Atlassian Support (post-migration) that contains a map of on-prem Confluence page IDs/short URLs -> corresponding URLs in Cloud. So you can skip ahead to "Putting the mapping table into use with Apache RewriteMap"

"Magic Redirector™"

Way back in 2019, @Ramona Scripcaru from Atlassian mentioned an undocumented feature of Confluence Cloud  that I am calling the "Confluence Page Title Magic Redirector™", or "Magic Redirector™" for short.

On Confluence Cloud every page URL includes the page Id, one that does NOT match what you had on-prem:

https://YOURCOMPANY.atlassian.net/wiki/spaces/INFO/pages/8721/Simple+Page+Title

But Ramona revealed that you can actually use a special URL that will get automatically redirected to that URL. Notably, it is a URL that does not contain a page Id:

https://YOURCOMPANY.atlassian.net/wiki/display/SPACEKEY/Page+Title

If you look closely, you'll see the format matches the same pattern as a non-pageId URL on Server/DC:

https://confluence.YOURCOMPANY.com/display/SPACEKEY/Page+Title

So then, it seems like all we need to do is ask NGINX or Apache point at the new host, with an extra /wiki/ added into the URL. Here's what those rules would look like:

NGINX

return 301 $scheme://YOURCOMPANY.atlassian.net/wiki$request_uri;

Apache

RedirectMatch 301 "^(.*)$" "https://YOURCOMPANY.atlassian.net/wiki$1"

But wait, you're not done.

You have to consider Page Ids

On Confluence Server/DC if a page title has certain punctuation marks in it (%&?/\;"§+:), then the page's URL looks like this, with a page Id:

https://confluence.YOURCOMPANY.com/pages/viewpage.action?pageId=123456

Otherwise, it looks like this, with the title right there in the URL:

https://confluence.YOURCOMPANY.com/display/INFO/Simple+Page+Title

This is all explained in Atlassian's documentation:

• The Differences Between Various URL Formats for a Confluence Page

The upshot is that if you have page titles with those characters, the URL for that page does not include the SPACEKEY and Page Title that the "Magic Redirector™" could use to then send users to the correct page. URLs for these pages just have a pageId.

Lookup page Ids

So what Ed figured out is that you can create a lookup table (using data extracted from SQL) to map a pageId to its SPACEKEY and Page Title, and then "Magic Redirector™" can do its thing.

Now Ed has amazing developers to write their own lookup tool+redirector in Java.

Mere mortals like me have to rely on existing tools, and luckily there is a handy Apache feature called RewriteMap that works perfectly for this.

Making a mapping table

So then, let's get into it. If you want to make a mapping table, you'll need the space keys, page Ids and titles from your Confluence Server/DC instance.

SQL script to get data

The fastest way to get this data is by connecting to the database directly. Here's a MySQL command that lets you get it or a specific space (great for testing):

SELECT CONTENTID, SPACEKEY, TITLE FROM CONTENT JOIN SPACES S on CONTENT.SPACEID = S.SPACEID WHERE CONTENTTYPE = 'PAGE' AND PREVVER IS NULL AND CONTENT_STATUS = 'current' AND S.SPACEKEY='INFO';

If you pipe that to a file, you should get something like this:

CONTENTID       SPACEKEY     TITLE
172376419       INFO         Slusho
176902673       INFO         Do Androids Dream of Electric Sheep?
176903761       INFO         Relocation Policy/Process
176903794       INFO         Story of Your Life
186251233       INFO         "Imagine If" Discovery Sessions
199307491       INFO         Talking Points
199308234       INFO         Eats; Shoots; Leaves
209349090       INFO         This + That

 So you need to turn that into a mapping table that looks like this:

176902673 /wiki/display/INFO/Do%2BAndroids%2BDream%2Bof%2BElectric%2BSheep%3F
199308234 /wiki/display/INFO/Eats%3B%2BShoots%3B%2BLeaves

PageId Mapping table script

I wrote a small Perl script to create the mapping table. To make the table smaller (which should make it faster), I omit pages whose titles do not have any punctuation marks, because their URLs should not require a lookup. (No page Id.) That's why the list gets shorter.

Script: https://github.com/darryllee/confredir/blob/main/pageidmap.pl

"Magic Redirector™" limitations and workaround

Oh, but there's one other hitch. The "Magic Redirector™" still doesn't like certain characters. In testing, I found that Confluence Cloud still does not like the following character in titles: & / + %

So a URL like this will NOT get properly redirected:

https://YOURCOMPANY.atlassian.net/wiki/display/INFO/Page+With+This&That+or+This/That

While one option would be go back to your Server/DC instance and rewrite all those titles to remove or replace the offending characters, I found another workaround - redirecting to a Search page with the title already entered. The URL (with encoding) ends up looking like this:
https://YOURCOMPANY.atlassian.net/wiki/search?text=Page%20With%20This%20%26%20That%20or%20This%2FThat

And so the user ends up on a page like this, which yes, requires a second click, but hopefully lands them on the right page:

So my Perl script also looks for & / + % in titles, and if found, redirects users to the Search page for that title. So they look like this:

176903761 /wiki/search?text=Relocation%20Policy%2FProcess
209349090 /wiki/search?text=This%20%2B%20That

Putting the mapping table into use with Apache RewriteMap

Here's the Apache rule that does the page Id to title mapping, using our table:

RewriteMap pageids "txt:/etc/httpd/pageids.txt"
RewriteCond %{REQUEST_URI} "^/pages/viewpage.action$"
RewriteCond %{QUERY_STRING} pageId=(.*)
RewriteRule . ${cloudhost}${pageids:%1|/wiki/spaces/TEST/pages/44073006/Migrated+Confluence+Page+Not+Found}? [L,R=301,NE]

RewriteMap defines your pageids lookup table and its location.

The first RewriteCond makes sure the original URL is actually looking trying to view a page by Id.

The second RewriteCond checks that there is a pageId specified, and stores it

The RewriteRule (which should all be on one line) replaces the original request with what is found when searching the lookup table. I created an alternative page, "Migrated Confluence Page Not Found", to catch any pages that may have been missed.

• L = Last rule, don't process any other ones after this one.
• R=301 means redirect this link permanently
• NE = No Escape - we are already encoding special characters in the lookup table, so we should not double-escape them.

Tiny Links

One last fun bit. Confluence has a thing called Tiny Links. The allow you to have "shorter" URLs that can be shared, that look like this:

https://confluence.YOURCOMPANY.com/x/Y0FGCg

Unfortunately, these links also change when you migrate from Server/DC to Cloud, so if you want old links to automatically redirect… you guessed it, you will need a redirector.

We basically use the same data as above, but we generate a mapping file for every page, because Tiny Links could be used for potentially any page. But how to generate Tiny Links? Well, Atlassian has a script for that: How to programmatically generate the tiny link of a Confluence page 

I adapted their script to read from the MySQL data, and then convert it using Atlassian's algorithm to produce a table like this (accounting for the characters that that "Magic Redirector™" cannot handle):

Y0FGCg /wiki/display/INFO/Slusho
EVKLCg /wiki/display/INFO/Do%2BAndroids%2BDream%2Bof%2BElectric%2BSheep%3F
UVaLCg /wiki/search?text=Relocation%20Policy%2FProcess
claLCg /wiki/display/INFO/Story%2Bof%2BYour%2BLife
4fcZCw /wiki/display/INFO/%22Imagine%2BIf%22%2BDiscovery%2BSessions
4zDhCw /wiki/display/INFO/Talking%2BPoints
yjPhCw /wiki/display/INFO/Eats%3B%2BShoots%3B%2BLeaves
4ml6DA /wiki/search?text=This%20%2B%20That

Script: https://github.com/darryllee/confredir/blob/main/tinylinkmap.pl

The Future

Well, last week, this bug was resolved:

• MIG-507 - Provide the corresponding URL information between Confluence Server and Confluence Cloud after migration

Comment from 02/Oct/2023:
"URL mappings before and after migration to cloud, including pageId changes can now be requested via Atlassian support as part of post migraiton link fixing: Links broken after Server to Cloud or Cloud to Cloud migration | Confluence | Atlassian Documentation"

Well that's pretty cool. So you could write a rewrite rule that maps every old link to the exact new link, based on a mapping table you would construct from the CSV data from Atlassian support. The mappings would just be a lot more accurate and you would not have to worry about the undocumented and tricky nature of the Magic Redirector™.

What about Jira?

On Jira, issue keys stay the same, so the rules would be much simpler (especially since Jira Cloud does not require anything after the hostname like Confluence Cloud's /wiki/)

NGINX

return 301 $scheme://YOURCOMPANY.atlassian.net$request_uri;

Apache

RedirectMatch 301 "^(.*)$" "https://YOURCOMPANY.atlassian.net/$1"

These rules would automatically redirect somebody going to an old Jira Server/DC link:

https://jira.YOURCOMPANY.com/browse/BUG-123

To the shiny new cloud site:

https://YOURCOMPANY.atlassian.net/browse/BUG-123

Nice, right? But wait, will this work for links to filters? What about boards?

Welp, all of the IDs for filters and boards change during migration, so old links like https://jira.YOURCOMPANY.com/secure/RapidBoard.jspa?rapidView=1157 will NOT work when translated to https://YOURCOMPANY.atlassian.net/secure/RapidBoard.jspa?rapidView=1157

Because the board that had an id of 1157 on Server/DC may now have an id of 38 on Cloud. Bummer.

Now it turns out that filter names are unique, so post-migration you could use the API to dump all the filter names and new IDs from the Cloud and use that to construct a mapping file, and maybe you can do this with boards too (although board names are NOT unique).

Reference:

Previous Discussions

• Post migration to the cloud - Redirection (2019-06-27)
• redirect from old confluence server URLs (2021-01-14)
• Redirecting from Server to Cloud - which URLs to handle? (2021-05-14)
• Redirecting page links from server to cloud after migration? (2021-05-25)
• Post-migration redirects? (2021-07-18)
• Migrating Atlassian Applications From Server to Cloud | Atlas Authority (2021-11-16)
• Is there a way to redirect users who browse to Jira/Confluence URLS after cloud migration? (2022-11-05)

Documentation on Confluence URL formats/Tiny Links

• The Differences Between Various URL Formats for a Confluence Page
• How to programmatically generate the tiny link of a Confluence page 

[Dave Liao]

@Darryl Lee - this is glorious!

p.s. If only we had a way to handle redirects from a Cloud to an on-prem migration? 🫠

p.s.s. I'll say, this article brings back PTSD of a time when we migrated from one software platform to another, many moons ago... but it's a good PTSD?

[Robert Wen_ReleaseTEAM_]

THis is amazing!

[Darryl Lee]

So I really wanted to do this with NGINX, since years ago I switched to it from Apache for reverse proxying Jira/Confluence.

But when I came to write this I couldn't wrap my head around their map directive. Luckily, Google found this: URL mapping with Nginx.

So I think we need:

map $request_uri $old_id {
  "~^/pages/viewpage.action?pageId=([0-9]+) $1;
}

map $old_id $new_path {
  include /etc/nginx/snippets/pageid.conf;
}

And rewritemap.conf would look very similar to our existing mapping file, with the addition of a trailing semicolon at the end of each line:

176903761 /wiki/search?text=Relocation%20Policy%2FProcess;
209349090 /wiki/search?text=This%20%2B%20That;

 And then finally we have this in the server block:

server {
if ($new_path) {
    return 301 https://YOURCOMPANY.atlassian.net$new_path;
  }
}

The article has some important details about tuning map_hash_max_size and map_hash_bucket_size settings so it runs as optimally as possible.

I would love if there are NGINX experts out there who could weigh-in on this. Or maybe I'll actually try testing it soon. :-}

[Matt Doar]

That's really useful to have all this documented, thank you!

However I'm not sure that filter names are unique except per user. Should work if you add the userid though.

[Dave Liao]

@Matt Doar - that's true, filter names aren't unique!

I think Darryl meant filter IDs are unique for each filter, in each environment. That'll allow for a mapping table to be made for any redirector.

[Darryl Lee]

Oof, yeah @Matt Doar is 100% right. I must've tested trying to create another filter of MY OWN with the same name. Doh.

So yeah, you'd need to generate a mapping file by ALSO looking at Owners, and OH FUN, GDPR means you'll have to map user-readable usernames to hashed user IDs.

So I found the Cloud Endpoint to get all filters:

https://YOURSITE.atlassian.net/rest/api/3/filter/search?expand=owner

But weirdly (and ANNOYINGLY) there doesn't seem to be an equivalent call for ON-PREM? This seems confirmed by Ian Ragudo back in 2019: https://community.developer.atlassian.com/t/get-rest-api-3-filter-search/29459/4

And ugh, 4 years later, there still does not seem to be a method to search/dump all filters:

https://docs.atlassian.com/software/jira/docs/api/REST/9.11.0/#api/2/filter

SO uh, I guess you could dump them via SQL. Awesome.

SELECT * from searchrequest

(Dump everything? Sure, why not!)

Oof, and uh, to get that usermapping, welp, if you're LUCKY your OLD usernames match say, the first part of email addresses, so darryllee@MYWORK.com -> darryllee, and so you could take your exported users file and do a bit of Excel stripping and make yourself a little lookup table to match on-prem darryllee that to the fun 123456:55fffff50-f179-4f17-f747-ff1484246f4f Cloud User Id.

BUT IF your old usernames don't match email (like if darryllee@MYWORK.com -> dlee) then UGH, I guess you then need to do a cross-check on Email address to map dlee to 123456:55fffff50-f179-4f17-f747-ff1484246f4f.

I leave that as an exercise for the reader. :-}

[Darryl Lee]

@Dave Liao redirecting from Cloud to On-Prem would be amazing, but it would likely have to be a service Atlassian owns/manages since they own the *.atlassian.net.

HOWEVER, with custom domains now a reality, I suppose you could extract all of your Cloud URLs from a backup file, OR hit the API. The tricky part would be figuring out how to map them to On-Prem URLs. I wonder if import logs contain that data.

[Darryl Lee]

So on a old, but related post, regarding using mapping data that you can request from Atlassian, @eric_gagnon_banq asks,

what could guide decision on using one or the other approach?

Maybe it's a late solution for those that have completed the migration already? Or the mapping file doesnt provide solution for tinylink?

Someone else is working on the migration, I'll try to see if I can get a sample of the mapping file generated by migration tool and see if I can figure this out.

Based on a comment from @Daniel Serkowski there are two ways to get mapping data. 

1. If you finished the links migration process (post-migration task) recently, our support team can assist you in exporting audit logs in CSV format (as this feature is not yet public). The exported CSV files will contain all changes made from old to new links. Here's a small sample of such a file: 

...

However, please be aware of the following limitations:

• We can only store this data for 29 days.

• Only links that have been migrated will be included.

I don't think that data is what we want, because the links migration tool does NOT cover links outside of Jira/Confluence: emails, docs, spreadsheets, presentations, Testrail, Salesforce, Zendesk, Internal apps.

Instead I think you would want to request this from support:

if it does not meet your requirements (for instance, if the link migration process was carried out a while ago), the support team can offer an alternative method of exporting mappings (focused on pageId/tinyurl values). The method is based on database queries, so will require extra consent from you, etc. The format for this is as follows:

"server_page_id","cloud_page_id","server_page_type","cloud_page_type","server_page_title","cloud_page_title","server_space_key","cloud_space_key","server_url","cloud_url","tiny_server_url","tiny_cloud_url"

[Example mapping file does include tiny links!]

2. Currently, we don't have a specific workflow for this. However, to expedite the process, you could refer to this ticket (https://jira.atlassian.com/browse/MIG-507) or link my comment there in the description of your support ticket.

SO yeah, I think it might be better to get a full mapping table from Atlassian. I'm actually really excited about this now, because at my day job we FINALLY got approved to go forward with a migration, and I'm going to get to try this for real. 

In answer to your question, I would use the table from Atlassian, because it will not involve any guessing. My mapping file will now have an exact ID to redirect people to, so the mapping file link to that instead of page title and MOST IMPORTANTLY you will not have to fuss with special characters and relying on the "Magic Redirector™". Instead you would just map old PageID to new PageID (with spacekey, which is also in the file from Atlassian):

176902673 /wiki/spaces/INFO/pages/835485712  

[eric_gagnon_banq]

Thanks for the answer about my question regarding the use of your solution vs what's offered by Atlassian.

[Darryl Lee]

One last clarification @eric_gagnon_banq - Atlassian doesn't provide a "solution", per se.

You're still going to be responsible for setting up the redirection service, using Apache or NGINX, etc.

What Atlassian can now provide is a CSV export of OLD and NEW page IDs and Tiny Links, which should make it possible to create a much more accurate mapping table than what I cobbled together.

But unfortunately it's still a bit of work. 

If Atlassian happens to be listening, what would be AMAZING is if they created an "upgrade" that replaced your on-prem Confluence with a redirector where you just need to upload the CSV you'd get from Support. It would listen the same port Confluence was on (8090) and serve up redirects based on the data in the CSV.

[eric_gagnon_banq]

"You're still going to be responsible for setting up the redirection service, using Apache or NGINX, etc."

Yes sure, I already understood that part.

[Adrien RESTAUT]

Hello @Darryl Lee 

So I read your message from the 17th of October related to nginx, and I made it work.
First, let me thank you for creating that page which is full of technical and useful infos.

Second, I'm no nginx expert, I spent several days on making the few lines in nginx.conf work...

Prepare the included pages

• For pageIds, you need to format the CSV (use Notepad++) as follows:

33839684 688135;
43811752 688141;

• For tinyURLs, like this:

~HoqcAg F4AK;
~I4qcAg GYAK;
~i4qcAg HYAK;
~noqcAg IYAK;

 --> I had to put ~ at the beginning of every lines because nginx map is not case-sensitive, and you see above that 2 lines are the same but a capital letter...

Use nginx map twice in http block

Mind that our Confluence Server was located in /Confluence/ behind our webserver.

• For pageIds:

http {
    map_hash_max_size 8192;
    map $request_uri $old_id {
       ~^/Confluence/pages/viewpage.action\?pageId=([0-9]+) $1;
       default $request_uri;
    }
    map $old_id $new_id {
       include /etc/nginx/snippets/rewritepageids.conf;
    }

--> The biggest difficutly I had is that you need to escape the ? above with a "\?".

• For tinyURLs:

    map $uri $old_tinyid {
      ~^/Confluence/x/(.+) $1;
      default $uri;
    }
    map $old_tinyid $new_tinyid {
      include /etc/nginx/snippets/rewritetinyurls.conf;
    }

Return conditionnally new URLs in location block

location /Confluence/ {
            if ($new_id) {
                return 301 https://customer.atlassian.net/wiki/pages/viewpage.action?pageId=$new_id;
            }
            if ($new_tinyid) {
                return 301 https://customer.atlassian.net/wiki/x/$new_tinyid;
            }
            rewrite ^/Confluence(.+)$ https://customer.atlassian.net/wiki$1 permanent;
}

I hope it works for you.

[Dam]

really nice one ;)