advice after anonymous access allowed rampant page and draft creation

[Andy Holt]

We have discovered a Confluence space that accidentally had the EDITSPACE permission allowed for anonymous users.  Doh.

Spammers found this and added >2000 pages.  This isn't so bad.

However, web crawlers did web crawler things, and hit the Create Page button approximately 750k times, mostly with URLs like this:

https://confluence.example.com:443/pages/createpage.action?fromPageId=240222349&spaceKey=EXT

We can see some of these pages, by using the ID in the URL, and they show with title $helper.page.title.

The overall size of our database (AWS RDS Aurora PostgreSQL) went from ~20 GB to ~110 GB due to this.

In particular, the confancestors table was previously around 87 MB and is now 84 GB (1000x times increase).  That table now has 777 million rows (was about 700k rows).

The space only had a few pages before this started, so I am happy to move those, and then blow away the space entirely.  However, on a test clone of our production system, I have initiated an entire space removal, which gets stuck at 2% with an ever-increasing 'Time Remaining'.  While this runs, the size of the confancestors table again grows rapidly (added another 30 GB when left running for 1 day), although the row count does not change.

Any advice on a better recovery / removal strategy ?

[Kristian Klima]

Hi @Andy Holt and sorry for liking your post. OMG....

Seriously, though.

To save your pages, you can use the customize option in space export menu - export just your pages as an XML file, and once you delete the space, reimport it back.

If space deletion doesn't work, you can try to delete the root page with the 'delete child pages option'. 

Then you can try using Confluence API and script your way out or you can try a marketplace app:

I found these resources...

• https://marketplace.atlassian.com/apps/1214087/space-admin-for-confluence?hosting=datacenter&tab=overview
• https://community.atlassian.com/t5/Confluence-articles/How-to-clean-programmatically-old-drafts-and-purge-trash-with/ba-p/876791
• https://stackoverflow.com/questions/43888882/update-delete-confluence-page-using-python-code

Back in day, after a huge migration of thousands of docs to a Confluence Server environment, the team I work with ran a cleanup and I'm pretty sure they did not do that manually (we're talking hundreds if not thousands of spaces).

EDIT: On Cloud, those drafts end up in 'undefined' section of the space settings (next to Archive, Trash...) I've not worked with server for over 2 years now but again, this might be a clue...

[Andy Holt]

Atlassian support have been very helpful (thank you Shekhar!).

Mostly, this has involved using resolution 2 from this KB article.  It's basically, stop Confluence, run a bunch of SQL queries to manually remove the space, clear out the node's index files, restart Confluence, let it automatically rebuild the index (I did not know it did this!) and lastly trigger the scheduled job to repair (rebuild in this case) the confancestors table.

Result, a much smaller, quicker, happier Confluence service.  But that's been two weeks' work for me! :)

[Kristian Klima]

YAY! Glad to hear that @Andy Holt .

Maybe it's time to look for a solution that will allow you to make your content public without opening your Confluence spaces to anonymous access :) 

[Andy Holt]

Anon read was always the intent in this space, just not anon edit! :-)

[Kristian Klima]

The road to hell hell is paved with good permissions intentions :) 

Humans see (a setting), humans do (play with the setting).

Stay safe!

[Nicolas Grossi]

@Andy Holt You can't recover a backup "before the attack" and change the edit permission on that space and put it back online ?

Nicolas

[Andy Holt]

Would be nice - but it was in Sept 2023. Doh (again).

[Darryl Lee]

Wait... wait. You have AWS RDS Aurora PostgreSQL.

Did you explicitly disable automated backups/snapshots? :-{

Oh wait... that's not possible. But I guess you could've reduced retention period to 1 day.

Their docs say:

• You can't disable automated backups on Aurora. The backup retention period for Aurora is managed by the DB cluster.

[Andy Holt]

No, I haven't disabled automated backups.  It's true that we don't have them going back that far, but what I meant was, there's no way we would wind back our Confluence that far (even if we had backups that old), and lose all the work between Sept 2023 and now across all spaces.

Thanks for reading my tale of woe though. :-)  I'm now getting help from Atlassian support with removing the space anyway.

[Darryl Lee]

Ohhhh, the incident occurred in Sept 2023 and you didn't notice the additional disk usage until now? Ouch.

I understand the problem now, and am glad to hear Atlassian is helping out.

[Andy Holt]

Interesting point.  Our disk monitoring has (so far!) focussed on things liable to run out of space.  Aurora DBs can grow without hitting a limit, so, well, yeah ...  We will be monitoring for unexpected growth from now on.  ;-)