问题
I have this code to crawl the details page
yield Request(flexibleItem[self.linkAttributeName],callback=self.parseDetails,dont_filter=True )
there is no error in the subURL because I tested it with the same method "GET"
I didn't get any error but simply python ignoring the callback function
It is a very huge project working on a server so I can't share the code . But here is the main architecture for what I am doing . Output is : in start request TRUE oooo
def start_requests(self):
print "in start request"
for url in self.start_urls:
yield Request (url, method='GET',dont_filter=True)
def parse(self, response):
"this method "
jsonResponse = json.loads(response.body_as_unicode())
flexibleItem = {}
flexibleItem['source'] = self.Source
flexibleItem['action'] = self.Action
flexibleItem['category'] = self.Category
jsonResponse = self.attributesXMLParser.correctResponse(jsonResponse)
flexibleItem[self.linkAttributeName] = self.attributesXMLParser.getPageLink(flexibleItem[self.linkAttributeName], flexibleItem)
meta = dict()
meta['flexibleItem'] = flexibleItem
if self.ParseDetailsPage=="TRUE" : "this value is from XML file"
print "TRUE"
yield Request(flexibleItem[self.linkAttributeName],callback=self.parseDetails,dont_filter=True)
print "oooooo"
else :
yield flexibleItem
def parseDetails(self, response):
print "in parse details "
jsonResponse = json.loads(response.body_as_unicode())
print jsonResponse
flexibleItem = {}
for key in self.Attributes:
flexibleItem[key]= .....bla bla bla
yield flexibleItem
来源:https://stackoverflow.com/questions/48844802/scrapy-doesnt-call-callback-function-even-with-no-filter