Get scrapy spider to crawl entire site

Question

I am using scrapy to crawl old sites that I own, I am using the code below as my spider. I don\'t mind having files outputted for each webpage, or a database with all the conten

Answer 1

To crawl whole site you should use the CrawlSpider instead of the scrapy.Spider

Here's an example

For your purposes try using something like this:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']

    rules = (
        Rule(LinkExtractor(), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)

Also, take a look at this article