Access django models inside of Scrapy

Question

Is it possible to access my django models inside of a Scrapy pipeline, so that I can save my scraped data straight to my model?

I\'ve seen this, but I don\'t really

Answer 1

Check out django-dynamic-scraper, it integrates a Scrapy spider manager into a Django site.

https://github.com/holgerd77/django-dynamic-scraper

Answer 2

Minor update to solve KeyError. Python(3)/Django(1.10)/Scrapy(1.2.0)

from django.core.management.base import BaseCommand

class Command(BaseCommand):    
    help = 'Scrapy commands. Accessible from: "Django manage.py". '

    def __init__(self, stdout=None, stderr=None, no_color=False):
        super().__init__(stdout=None, stderr=None, no_color=False)

        # Optional attribute declaration.
        self.no_color = no_color
        self.stderr = stderr
        self.stdout = stdout

        # Actual declaration of CLI command
        self._argv = None

    def run_from_argv(self, argv):
        self._argv = argv
        self.execute(stdout=None, stderr=None, no_color=False)

    def handle(self, *args, **options):
        from scrapy.cmdline import execute
        execute(self._argv[1:])

The SCRAPY_SETTINGS_MODULE declaration is still required.

os.environ.setdefault('SCRAPY_SETTINGS_MODULE', 'scrapy_project.settings')