Scrapy SgmlLinkExtractor question

Question

I am trying to make the SgmlLinkExtractor to work.

This is the signature:

SgmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), res         


        

Answer 1

if you check documentation a "Warning" is clearly written

"When writing crawl spider rules, avoid using parse as callback, since the Crawl Spider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work."

url for verification

Answer 2

allow=(r'/aadler/', ...

Answer 3

You are missing comma after first element for "rules" to be a tuple..

rules = (Rule(SgmlLinkExtractor(allow=('/careers/n.\w+', )), callback='parse', follow=True),)

Answer 4

You are overriding the "parse" method it appears. "parse", is a private method in CrawlSpider used to follow links.