How do I automate navigation to a website that requires authentication?

问题

Here's what I'm trying to achieve. I would like to write a script that will navigate to a website that requires me to be authenticated as myself, say Facebook, Live Spaces, Twitter or any other, and then have that script search for certain information on one of the pages of the website.

I've done something similar in the past with the Windows.Forms WebBrowser control, which is a full blown implementation of IE that can be controlled through code and will store whatever cookies you get once you're authenticated, but it was very unfriendly to modify and I was hoping to use a scripting language instead, maybe Powershell or something of that sort.

Are there maybe some good tutorials about this out there on the web?

Thanks!

回答1:

If Python is your first choice, Mechanize + Twill + Beautiful Soup.
Have a look to SO screen-scraping tag too.

回答2:

There are many alternatives. In the past I have automated my browser around authenticated sites with Chickenfoot, or from the console with Mechanize.

Then crawl the site and examine each pages HTML for your data.

回答3:

You can use IRobotSoft web scraper. It automates an IE WebBrowser control for web data mining. See their forum here http://www.irobotsoft.org/bb/ for typical questions.

回答4:

I have created a new PowerShell module, currently in Technology Preview. Try http://facebookpsmodule.codeplex.com, and let me know how this works for you!

来源：https://stackoverflow.com/questions/2598453/how-do-i-automate-navigation-to-a-website-that-requires-authentication