I wrote a basic crawler in Python with Scrapy. The purpose of it - to check broken links (404) on website.
But this crawler is not showing me every URL from website, as I
