How to bypass a 'cookiewall' when using scrapy?

问题

I'm a new user to Scrapy. After following the tutorials for extracting data from websites, I am trying to accomplish something similar on forums.

What I want is to extract all posts on a forum page (to start with). However, this particular forum has a 'cookie wall'. So when I want to extract from http://forum.fok.nl/topic/2413069, each session I first need to click the "Yes, I accept cookies"-button.

My very basic scraper currently looks like this:

class FokSpider(scrapy.Spider):
name = 'fok'
allowed_domains = ['forum.fok.nl']
start_urls = ['http://forum.fok.nl/']

def parse(self,response):
    divs = response.xpath("//div").extract()
    yield {'divs': divs}
    pass

The divs I get are not from the actual forum thread, but from the cookie wall.

Here's the html of the button:

<a href="javascript:acceptCookies()" class="button acc CookiesOK" onclick="document.forms['cookies'].submit();acceptCookies();">Ja, Ik wil een goed werkende site...<span class="smaller">...en accepteer de cookies</span></a>

Can anyone point me in the right direction on how to bypass this cookiewall (artificially 'click' the button) and go to the actual webpage I'm trying to scrape? (Even the right Google search terms/documentation pages etc would be very helpful)

回答1:

In the end I found multiple ways to solve this problem:

• Simply having adding /?token=77c1f767bc31859fee1ffe041343fa48&allowcookies=ACCEPTEER+ALLE+COOKIES to the start url worked for this specific case
• I later switched to a CrawlSpider instead of a normal Spider, then I could add the xpath of the cookie button as the first rule.
• Clicking the button using the earlier mentioned Selenium also worked, but is a lot of hassle that is not really necessary...

来源：https://stackoverflow.com/questions/47651881/how-to-bypass-a-cookiewall-when-using-scrapy