Scrapy - Grab all product details

问题

I need to grab all Product Details (with green tickmarks) from this page: https://sourceforge.net/software/product/Budget-Maestro/

    divs = response.xpath("//section[@class='row psp-section m-section-comm-details m-section-emphasized grey']/div[@class='list-outer column']/div")
    for div in divs:
        detail = div.xpath("./h3/text()").extract_first().strip() + ":"
        if detail!="Company Information:":
            divs2 = div.xpath(".//div[@class='list']/div")
            for div2 in divs2:
                dd = [val for val in div2.xpath("./text()").extract() if val.strip('\n').strip().strip('\n')]
                for d in dd:
                    detail = detail + d + ","
            detail = detail.strip(",")
            product_details = product_details + detail + "|"
    product_details = product_details.strip("|")

But it gives me some features with \n as well. And I'm sure there must be a smarter & shorter way to do this.

回答1:

If you need data only from "Product Details", check this:

In [6]: response.css("section.m-section-comm-details div.list svg").xpath('.//following-sibling::text()').extract()
Out[6]: 
[u' SaaS\n                        ',
 u' Windows\n                        ',
 u' Live Online ',
 u' In Person ',
 u' Online ',
 u' Business Hours ']

回答2:

Use this,

divs = [div.strip() for div in response.xpath('//*[contains(@class, "has-feature")]/text()').extract() if div.strip()]

Now Div is

[u'Accounts Payable', u'Accounts Receivable', u'Cash Management', u'General Ledger', u'Payroll', u'Project Accounting', u'"What If" Scenarios', u'Balance Sheet', u'Capital Asset Planning', u'Cash Management', u'Consolidation / Roll-Up', u'Forecasting', u'General Ledger', u'Income Statements', u'Multi-Company', u'Multi-Department / Project', u'Profit / Loss Statement', u'Project Budgeting', u'Run Rate Tracking', u'Version Control',u'"What If" Scenarios', u'Balance Sheet', u'Cash Management', u'Consolidation / Roll-Up', u'Forecasting', u'General Ledger', u'Income Statements', u'Profit / Loss Statement']

And i hope this is all you want. Iterate over this list now and do you logic :)

来源：https://stackoverflow.com/questions/54104633/scrapy-grab-all-product-details