How to properly iterate over unicode characters in Python

放肆的年华 提交于 2019-12-02 07:01:49

问题


I would like to iterate over a string and output all emojis.

I'm trying to iterate over the characters, and check them against an emoji list.

However, python seems to split the unicode characters into smaller ones, breaking my code. Example:

>>> list(u'Test \U0001f60d')
[u'T', u'e', u's', u't', u' ', u'\ud83d', u'\ude0d']

Any ideas why u'\U0001f60d' gets split?

Or what's a better way to extract all emojis? This was my original extraction code:

def get_emojis(text):
  emojis = []
  for character in text:
    if character in EMOJI_SET:
      emojis.append(character)
  return emojis

回答1:


Python pre-3.3 uses UTF-16LE (narrow build) or UTF-32LE (wide build) internally for storing Unicode, and due to leaky abstraction exposes this detail to the user. UTF-16LE uses surrogate pairs to represent Unicode characters above U+FFFF as two codepoints. Either use a wide Python build or switch to Python 3.3 or later to fix the issue.

One way of dealing with a narrow build is to match the surrogate pairs:

Python 2.7 (narrow build):

>>> s = u'Test \U0001f60d'
>>> len(s)
7
>>> re.findall(u'(?:[\ud800-\udbff][\udc00-\udfff])|.',s)
[u'T', u'e', u's', u't', u' ', u'\U0001f60d']

Python 3.6:

>>> s = 'Test \U0001f60d'
>>> len(s)
6
>>> list(s)
['T', 'e', 's', 't', ' ', '😍']



回答2:


Try this,

import re
re.findall(r'[^\w\s,]', my_list[0])

The regex r'[^\w\s,]' matches any character that is not a word, whitespace or comma.




回答3:


The problem is as described above. The possible actions to solve it described here



来源:https://stackoverflow.com/questions/46711888/how-to-properly-iterate-over-unicode-characters-in-python

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!