Scrapy spider not found error

Question

This is Windows 7 with python 2.7

I have a scrapy project in a directory called caps (this is where scrapy.cfg is)

My spider is located in caps\\caps\\spiders\\c

Answer 1

Have you set up the SPIDER_MODULES setting?

SPIDER_MODULES

Default: []

A list of modules where Scrapy will look for spiders.

Example:

SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev']

Answer 2

Also, it is possible that you have not deployed your spider. SO first use "scrapyd" to up the server and then use "scrapyd-deploy" to deploy and then run the command.

Answer 3

Sometime this strange behaviour is caused by LOG_STDOUT = True

It defaults to False though, so check it and if it is set to True - try to set it to default

LOG_STDOUT = False

This is a logged issue

Answer 4

just to add my learning point here.

I had my crawler working, it suddenly started giving the error and came here to find the solution.

couldn't fix it, so checked my changes and i stupidly created a new variable "name".

This causes scrapy to not find the spider name also.

Answer 5

For anyone who might have the same problem, not only you need to set the name of the spider and check for SPIDER_MODULES and NEWSPIDER_MODULE in your scrapy settings, if you are running a scrapyd service, you also need to restart in order to apply any change you have made

Answer 6

I fixed by fixing my filename.

Originally, my.spider.py. Fixed, myspider.py.

I'm very new to python and scrapy so I'm not sure if this is a dumb mistake on my part.