How to crawl data from the linked webpages on a webpage we are crawling

问题

I am crawling the names of the colleges on this webpage, but, i also want to crawl the number of faculties in these colleges which is available if open the specific webpages of the colleges by clicking the name of the college.

What should i append to this code to get the result. The result should be in the form of [(name1, faculty1), (name2,faculty2),... ]

import scrapy
class QuotesSpider(scrapy.Spider):
    name = "student"
    start_urls = [
        'http://www.engineering.careers360.com/colleges/list-of-engineering-colleges-in-karnataka?sort_filter=alpha',
    ]

    def parse(self, response):
        for students in response.css('li.search-result'):
            yield {
                'name': students.css('div.title a::text').extract(),                   
            }

回答1:

import scrapy
class QuotesSpider(scrapy.Spider):
    name = "student"
    start_urls = [
        'http://www.engineering.careers360.com/colleges/list-of-engineering-colleges-in-karnataka?sort_filter=alpha',
    ]

    def parse(self, response):
        for students in response.css('li.search-result'):
            req = scrapy.Request(students.css(SELECT_URL), callback=self.parse_student)
            req.meta['name'] = students.css('div.title a::text').extract()
            yield req

    def parse_student(self, response):
        yield {
            'name': response.meta.get('name')
            'other data': response.css(SELECTOR)
        }

Should be something like this. So you send the name of the student in the meta data of the request. That allows you to request it in your next request.

If the data is also available on the last page you scrape in parse_student you might want to consider not sending it in the meta data but just to scrape it from the last page.

来源：https://stackoverflow.com/questions/44476674/how-to-crawl-data-from-the-linked-webpages-on-a-webpage-we-are-crawling