scrapy crawl [spider-name] fault

Question

Hi guys i am building a web scraping project using scrapy framework and python. In spider folder of my project i have two spiders named spider1 and spider2

spider1.p

Answer 1

I tackled the same problem, however removing all *.pyc files from everywhere in my project did the job.

Especially I think settings.pyc is important to remove.

Hope that helps.

Answer 2

Building on Nomad's answer. You can avoid the creation of all but one pyc file during development by adding:

import sys
sys.dont_write_bytecode = True

to the project's "__init__.py" file.

This will prevent .pyc files from being created. Especially useful if you are working on a project and you rename the file name of a spider. Prevents the cached pyc of the old spiders remaining, and a few other gotchas.