问题
I'm trying to do a HTTP GET on craigslist sfbay.craigslist.org. Here is my (ruby) code which is really simple
require 'net/http'
result = Net::HTTP.get(URI.parse('http://sfbay.craigslist.org'))
I end up getting an error "This IP has been automatically blocked."
This behaviour only happens when I try this from Amazon EC2 or on heroku. When I try again on my own computer localhost I get the correct result. Does this have to do with Amazon EC2?
I'm wondering if other people have had the same issue. What can I do to access craigslist from EC2?
回答1:
I can confirm that Craigslist is blocking from the major Amazon EC2 IP ranges by IP (not by user agent). It works elsewhere, though I suspect any volume would cause other IPs to get blocked.
You could step around it with tor. More significantly, this stackoverflow question discusses data sources used by craigslist mashups.
I even tested a Brazil EC2, assuming they might not have all the CIDRs blocked. No bueno.
来源：https://stackoverflow.com/questions/14328955/http-get-on-craigslist-blocked