How to scrape the pages requiring login?

I am trying to crawl through the wiki pages but the authentication required is not allowing me to get the form field names or anything. I was wondering if there is a better way to do this. Thanks!

 

1 answer

0 vote

If the web interface wants you to authenticate (to login), as that page does not allow anonymous access, then your web scraper also needs to authenticate (as that is essentially doing the same thing: getting an HTTP response about that Confluence page). 

I suggest you follow the Confluence REST API way, but even in that case: you need to authenticate. 

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Posted Jul 10, 2018 in Confluence

We want to see the templates you've created in Confluence!

Hi Community, Jessica here from the Confluence Product Marketing team!  July’s community challenge is all about sharing pictures  — and as an extension of our first post on what ...

915 views 23 12
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you