Convert XML/HTML Entities into Unicode String in Python [duplicate]

前端未结

关注

 10  2228

醉话见心

相关标签:

10条回答

春和景丽

2020-11-22 10:28
Another solution is the builtin library xml.sax.saxutils (both for html and xml). However, it will convert only &gt, &amp and &lt.
```
from xml.sax.saxutils import unescape

escaped_text = unescape(text_to_escape)
```
0 讨论(0)
发布评论:

提交评论
- 加载中...

你的背包

2020-11-22 10:31

You could find an answer here -- Getting international characters from a web page?

EDIT: It seems like BeautifulSoup doesn't convert entities written in hexadecimal form. It can be fixed:

import copy, re
from BeautifulSoup import BeautifulSoup

hexentityMassage = copy.copy(BeautifulSoup.MARKUP_MASSAGE)
# replace hexadecimal character reference by decimal one
hexentityMassage += [(re.compile('&#x([^;]+);'), 
                     lambda m: '&#%d;' % int(m.group(1), 16))]

def convert(html):
    return BeautifulSoup(html,
        convertEntities=BeautifulSoup.HTML_ENTITIES,
        markupMassage=hexentityMassage).contents[0].string

html = '<html>&#x01ce;&#462;</html>'
print repr(convert(html))
# u'\u01ce\u01ce'

EDIT:

unescape() function mentioned by @dF which uses htmlentitydefs standard module and unichr() might be more appropriate in this case.

0 讨论(0)

-上瘾入骨i

2020-11-22 10:36

The standard lib’s very own HTMLParser has an undocumented function unescape() which does exactly what you think it does:

up to Python 3.4:

import HTMLParser
h = HTMLParser.HTMLParser()
h.unescape('&copy; 2010') # u'\xa9 2010'
h.unescape('&#169; 2010') # u'\xa9 2010'

Python 3.4+:

import html
html.unescape('&copy; 2010') # u'\xa9 2010'
html.unescape('&#169; 2010') # u'\xa9 2010'

0 讨论(0)

星月不相逢

2020-11-22 10:37
If you are on Python 3.4 or newer, you can simply use the html.unescape:
```
import html

s = html.unescape(s)
```
0 讨论(0)
发布评论:

提交评论
- 加载中...
予麋鹿

2020-11-22 10:38
An alternative, if you have lxml:
```
>>> import lxml.html
>>> lxml.html.fromstring('&#x01ce').text
u'\u01ce'
```
0 讨论(0)
发布评论:

提交评论
- 加载中...

离开以前

2020-11-22 10:38

This is a function which should help you to get it right and convert entities back to utf-8 characters.

def unescape(text):
   """Removes HTML or XML character references 
      and entities from a text string.
   @param text The HTML (or XML) source text.
   @return The plain text, as a Unicode string, if necessary.
   from Fredrik Lundh
   2008-01-03: input only unicode characters string.
   http://effbot.org/zone/re-sub.htm#unescape-html
   """
   def fixup(m):
      text = m.group(0)
      if text[:2] == "&#":
         # character reference
         try:
            if text[:3] == "&#x":
               return unichr(int(text[3:-1], 16))
            else:
               return unichr(int(text[2:-1]))
         except ValueError:
            print "Value Error"
            pass
      else:
         # named entity
         # reescape the reserved characters.
         try:
            if text[1:-1] == "amp":
               text = "&amp;amp;"
            elif text[1:-1] == "gt":
               text = "&amp;gt;"
            elif text[1:-1] == "lt":
               text = "&amp;lt;"
            else:
               print text[1:-1]
               text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
         except KeyError:
            print "keyerror"
            pass
      return text # leave as is
   return re.sub("&#?\w+;", fixup, text)

0 讨论(0)

1 2 下一页

热议问题