Scrapy doesn't call callback function even with no filter

老子叫甜甜 提交于 2020-01-25 09:13:16

问题


I have this code to crawl the details page

yield Request(flexibleItem[self.linkAttributeName],callback=self.parseDetails,dont_filter=True  )

there is no error in the subURL because I tested it with the same method "GET"

I didn't get any error but simply python ignoring the callback function

It is a very huge project working on a server so I can't share the code . But here is the main architecture for what I am doing . Output is : in start request TRUE oooo

def start_requests(self):
    print "in start request"
    for url in self.start_urls:
        yield Request (url, method='GET',dont_filter=True)

def parse(self, response):
    "this method "
    jsonResponse = json.loads(response.body_as_unicode())
    flexibleItem = {}
    flexibleItem['source'] = self.Source
    flexibleItem['action'] = self.Action
    flexibleItem['category'] = self.Category
    jsonResponse = self.attributesXMLParser.correctResponse(jsonResponse)

    flexibleItem[self.linkAttributeName] = self.attributesXMLParser.getPageLink(flexibleItem[self.linkAttributeName], flexibleItem)
    meta = dict()
    meta['flexibleItem'] = flexibleItem
    if self.ParseDetailsPage=="TRUE" : "this value is from XML file"
         print "TRUE"
         yield Request(flexibleItem[self.linkAttributeName],callback=self.parseDetails,dont_filter=True)
         print "oooooo"
    else :
         yield flexibleItem

def parseDetails(self, response):
    print "in parse details "
    jsonResponse = json.loads(response.body_as_unicode())
    print jsonResponse
    flexibleItem = {}
    for key in self.Attributes:
       flexibleItem[key]= .....bla bla bla
    yield flexibleItem

来源:https://stackoverflow.com/questions/48844802/scrapy-doesnt-call-callback-function-even-with-no-filter

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!