Why Scrapy returns an Iframe?

Question

i want to crawl this site by Python-Scrapy

i try this

class Parik(scrapy.Spider):
    name = \"ooshop\"
    allowed_domains = [\"http://www.ooshop.co         


        

Answer 1

The website is protected by Incapsula, a website security service. It's providing your "browser" with a challenge that it must perform before being given a special cookie that gives you access to the website itself.

Fortunately, it's not that hard to bypass. Install incapsula-cracker and install its downloader middleware:

DOWNLOADER_MIDDLEWARES = {
    'incapsula.IncapsulaMiddleware': 900
}