How to scrape the pages requiring login?

[Kamalanathan S G]

I am trying to crawl through the wiki pages but the authentication required is not allowing me to get the form field names or anything. I was wondering if there is a better way to do this. Thanks!

[Aron Gombas _Midori_]

If the web interface wants you to authenticate (to login), as that page does not allow anonymous access, then your web scraper also needs to authenticate (as that is essentially doing the same thing: getting an HTTP response about that Confluence page). 

I suggest you follow the Confluence REST API way, but even in that case: you need to authenticate.