Scrapy start_urls

Question

The script (below) from this tutorial contains two start_urls.

from scrapy.spider import Spider
from scrapy.selector import Selector

from dirb         


        

Answer 1

If you use BaseSpider, inside the callback, you have to extract out your desired urls yourself and return a Request object.

If you use CrawlSpider, links extraction would be taken care of by the rules and the SgmlLinkExtractor associated with the rules.