Scrapy spider not found error

Question

This is Windows 7 with python 2.7

I have a scrapy project in a directory called caps (this is where scrapy.cfg is)

My spider is located in caps\\caps\\spiders\\c

Answer 1

make sure that your spider file is saved in your spider directory. the Crawler looks for the spider name in the spider directory

Answer 2

In my case, i set 'LOG_STDOUT=True', and scrapyd can not return the results to json response when you are looking for your spiders with '/listspiders.json'. And instead of that, the results are being printed to the log files you set at scrapyd's default_scrapyd.conf file. So, I changed the settings as this, and it worked well.

LOG_STDOUT = False

Answer 3

If you are following the tutorial from https://docs.scrapy.org/en/latest/intro/tutorial.html

Then do something like:

$ sudo apt install python-pip
$ pip install Scrapy
(logout, login)
$ cd
$ scrapy startproject tutorial
$ vi ~/tutorial/tutorial/spiders/quotes_spider.py
$ cd ~/tutorial/tutorial
$ scrapy crawl quotes

The error happens if you try to create the spiders directory yourself under ~/tutorial