Write functions for all scrapy spiders

问题

So I'm trying to write functions that can be called upon from all scrapy spiders. Is there one place in my project where I can just define these functions or do I need to import them in each spider?

Thanks

回答1:

You can't implicitly import code (at least not without hacking around) in python, after all explicit is better than implicit - so it's not a good idea.

However in scrapy it's very common to have base Spider class with common functions and methods.

Lets assume you have this tree:

├── myproject
│   ├── __init__.py
│   ├── spiders
│   │   ├── __init__.py
│   │   ├── spider1.py
│   │   ├── spider2.py
├── scrapy.cfg

We can create a base spider in spiders/__init__.py:

class BaseSpider(Spider):
    def common_parse(self, response):
        # do something     

And inherit from it in your spiders:

from myproject.spiders import BaseSpider
class Spider1(BaseSpider):
    def parse(self, response):
        # use common methods!
        if 'indicator' in response.body:
            self.common_parse(response)

来源：https://stackoverflow.com/questions/44198404/write-functions-for-all-scrapy-spiders