Why I am Getting '_SIGCHLDWaker' object has no attribute 'doWrite' in Scrapy?

Question

I am using Scrapy spiders inside Celery and I am getting this kind of errors randomly

Unhandled Error
Traceback (most recent call last):
  File \"/usr/lib/python         


        

Answer 1

I had the same issue. I'm working within a complex application, using asyncio, multiprocessing, Twisted and Scrapy all together.

The solution for me was to use asyncioreactor, by installing the alternate reactor before any imports in scrapy:

from twisted.internet import asyncioreactor
asyncioreactor.install()

from scrapy import stuff
# ...