发表新帖
发表新帖

Parsing HTML using Python

前端 未结
关注
7 627
爱一瞬间的悲伤
爱一瞬间的悲伤 2020-11-22 00:35

I\'m looking for an HTML Parser module for Python that can help me get the tags in the form of Python lists/dictionaries/objects.

If I have a document of the form:

相关标签:
7条回答
  • 野趣味
    2020-11-22 01:12

    I guess what you're looking for is pyquery:

    pyquery: a jquery-like library for python.

    An example of what you want may be like:

    from pyquery import PyQuery    
html = # Your HTML CODE
pq = PyQuery(html)
tag = pq('div#id') # or     tag = pq('div.class')
print tag.text()

    And it uses the same selectors as Firefox's or Chrome's inspect element. For example:

    the element selector is 'div#mw-head.noprint'

    The inspected element selector is 'div#mw-head.noprint'. So in pyquery, you just need to pass this selector:

    pq('div#mw-head.noprint')
    0 讨论(0)
  • 南旧
    2020-11-22 01:15

    Here you can read more about different HTML parsers in Python and their performance. Even though the article is a bit dated it still gives you a good overview.

    Python HTML parser performance

    I'd recommend BeautifulSoup even though it isn't built in. Just because it's so easy to work with for those kinds of tasks. Eg:

    import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen('http://www.google.com/')
soup = BeautifulSoup(page)

x = soup.body.find('div', attrs={'class' : 'container'}).text
    0 讨论(0)
  • 迷失自我
    2020-11-22 01:17

    I recommend lxml for parsing HTML. See "Parsing HTML" (on the lxml site).

    In my experience Beautiful Soup messes up on some complex HTML. I believe that is because Beautiful Soup is not a parser, rather a very good string analyzer.

    0 讨论(0)
  • 难免孤独
    2020-11-22 01:17

    I would use EHP

    https://github.com/iogf/ehp

    Here it is:

    from ehp import *

doc = '''<html>
<head>Heading</head>
<body attr1='val1'>
    <div class='container'>
        <div id='class'>Something here</div>
        <div>Something else</div>
    </div>
</body>
</html>
'''

html = Html()
dom = html.feed(doc)
for ind in dom.find('div', ('class', 'container')):
    print ind.text()

    Output:

    Something here
Something else
    0 讨论(0)
  • 遇见更好的自我
    2020-11-22 01:18

    I recommend using justext library:

    https://github.com/miso-belica/jusText

    Usage: Python2:

    import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print paragraph.text

    Python3: 

    import requests
import justext

response = requests.get("http://bbc.com/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print (paragraph.text)
    0 讨论(0)
  • 一整个雨季
    2020-11-22 01:19

    Compared to the other parser libraries lxml is extremely fast:

    • http://blog.dispatched.ch/2010/08/16/beautifulsoup-vs-lxml-performance/
    • http://www.ianbicking.org/blog/2008/03/python-html-parser-performance.html

    And with cssselect it’s quite easy to use for scraping HTML pages too:

    from lxml.html import parse
doc = parse('http://www.google.com').getroot()
for div in doc.cssselect('a'):
    print '%s: %s' % (div.text_content(), div.get('href'))

    lxml.html Documentation

    0 讨论(0)
 看不清?
提交回复
热议问题