Scrapy spider not found error

Question

This is Windows 7 with python 2.7

I have a scrapy project in a directory called caps (this is where scrapy.cfg is)

My spider is located in caps\\caps\\spiders\\c

Answer 1

Name attribute in CrawlSpider class defines the spider name and this name is used in command line for calling the spider to work.

import json

from scrapy import Spider
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.linkextractor import LinkExtractor

class NameSpider(CrawlSpider):
    name = 'name of spider'
    allowed_domains = ['allowed domains of web portal to be scrapped']
    start_urls = ['start url of of web portal to be scrapped']

    custom_settings = {
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
    }

    product_css = ['.main-menu']
    rules = [
        Rule(LinkExtractor(restrict_css=product_css), callback='parse'),
    ]

    def parse(self, response):
        //implementation of business logic