The Atlassian Community Forums are currently in read-only mode. We will be relaunching on a new platform on September 22 (read more here). We apologize for the extended downtime. For concerns or questions, please email communitymanagers@atlassian.com. See you on the other side, on the new Atlassian Community Forums! :)

×

Forums

Articles
Create
cancel
Showing results for 
Search instead for 
Did you mean: 

How to scrape the pages requiring login?

Kamalanathan S G
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
June 16, 2017

I am trying to crawl through the wiki pages but the authentication required is not allowing me to get the form field names or anything. I was wondering if there is a better way to do this. Thanks!

 

1 answer

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

0 votes
Aron Gombas _Midori_
Community Champion
June 19, 2017

If the web interface wants you to authenticate (to login), as that page does not allow anonymous access, then your web scraper also needs to authenticate (as that is essentially doing the same thing: getting an HTTP response about that Confluence page). 

I suggest you follow the Confluence REST API way, but even in that case: you need to authenticate. 

TAGS
AUG Leaders

Atlassian Community Events