Intermittent “getrandom() initialization failed” using scrapy spider

Question

I built a scrapy spider (scrapy 1.4). This spider is triggered on demand from a django website through django-rq and supervisord.

Here is the supervisord job that is

Answer 1

I had similar error, but with python-requests library:

Error([('', 'osrandom_rand_bytes', 'getrandom() initialization failed.')],)

This was caused by random number generator that failed to gather enough entropy in time. I've installed rng-tools and it solved the problem.