问题
I am trying to parse utf-8 strings into "bite sized" segments. For example, I would like to break down a text into "sentences".
Is there a comprehensive collection of characters (or regex) that correspond to end of sentences in all languages? I'm looking for something that would capture the Latin period, exclamation and interrogation marks, the Chinese and Japanese full stop, etc.
Something like the above but for the equivalent of a comma would be great too.
回答1:
I haven’t encountered any compilations of such information, and I would expect it to be a major effort to collect it. For some widely used languages, you could get the information from The Chicago Manual of Style. There is some information about punctuation marks commonly used in different languages at http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters-other.html but is covers just a small set of languages and does not distinguish sentence-terminating characters.
Using just characters won’t be enough, since e.g. in English, the full stop “.” occurs in many contexts where it does not terminate a sentence, as in “e.g.” or in “1.5”.
回答2:
You need to look at code points with the \p{Sentence_Break=STerm} or \p{Sentence_Break=ATerm} properties that also have the \p{Terminal_Punctuation} property. Running the unichars script against Unicode v6.1, we learn that these code points meet all those criteria:
$ unichars -gas '[\p{Sentence_Break=STerm}\p{Sentence_Break=ATerm}]' '\p{Terminal_Punctuation}'
U+00021 ! GC=Po SC=Common EXCLAMATION MARK
U+0002E . GC=Po SC=Common FULL STOP
U+0003F ? GC=Po SC=Common QUESTION MARK
U+00589 ։ GC=Po SC=Common ARMENIAN FULL STOP
U+0061F ؟ GC=Po SC=Common ARABIC QUESTION MARK
U+006D4 ۔ GC=Po SC=Arabic ARABIC FULL STOP
U+00700 ܀ GC=Po SC=Syriac SYRIAC END OF PARAGRAPH
U+00701 ܁ GC=Po SC=Syriac SYRIAC SUPRALINEAR FULL STOP
U+00702 ܂ GC=Po SC=Syriac SYRIAC SUBLINEAR FULL STOP
U+007F9 ߹ GC=Po SC=Nko NKO EXCLAMATION MARK
U+00964 । GC=Po SC=Common DEVANAGARI DANDA
U+00965 ॥ GC=Po SC=Common DEVANAGARI DOUBLE DANDA
U+0104A ၊ GC=Po SC=Myanmar MYANMAR SIGN LITTLE SECTION
U+0104B ။ GC=Po SC=Myanmar MYANMAR SIGN SECTION
U+01362 ። GC=Po SC=Ethiopic ETHIOPIC FULL STOP
U+01367 ፧ GC=Po SC=Ethiopic ETHIOPIC QUESTION MARK
U+01368 ፨ GC=Po SC=Ethiopic ETHIOPIC PARAGRAPH SEPARATOR
U+0166E ᙮ GC=Po SC=Canadian_Aboriginal CANADIAN SYLLABICS FULL STOP
U+01803 ᠃ GC=Po SC=Common MONGOLIAN FULL STOP
U+01809 ᠉ GC=Po SC=Mongolian MONGOLIAN MANCHU FULL STOP
U+01944 ᥄ GC=Po SC=Limbu LIMBU EXCLAMATION MARK
U+01945 ᥅ GC=Po SC=Limbu LIMBU QUESTION MARK
U+01AA8 ᪨ GC=Po SC=Tai_Tham TAI THAM SIGN KAAN
U+01AA9 ᪩ GC=Po SC=Tai_Tham TAI THAM SIGN KAANKUU
U+01AAA ᪪ GC=Po SC=Tai_Tham TAI THAM SIGN SATKAAN
U+01AAB ᪫ GC=Po SC=Tai_Tham TAI THAM SIGN SATKAANKUU
U+01B5A ᭚ GC=Po SC=Balinese BALINESE PANTI
U+01B5B ᭛ GC=Po SC=Balinese BALINESE PAMADA
U+01B5E ᭞ GC=Po SC=Balinese BALINESE CARIK SIKI
U+01B5F ᭟ GC=Po SC=Balinese BALINESE CARIK PAREREN
U+01C3B ᰻ GC=Po SC=Lepcha LEPCHA PUNCTUATION TA-ROL
U+01C3C ᰼ GC=Po SC=Lepcha LEPCHA PUNCTUATION NYET THYOOM TA-ROL
U+01C7E ᱾ GC=Po SC=Ol_Chiki OL CHIKI PUNCTUATION MUCAAD
U+01C7F ᱿ GC=Po SC=Ol_Chiki OL CHIKI PUNCTUATION DOUBLE MUCAAD
U+0203C ‼ GC=Po SC=Common DOUBLE EXCLAMATION MARK
U+0203D ‽ GC=Po SC=Common INTERROBANG
U+02047 ⁇ GC=Po SC=Common DOUBLE QUESTION MARK
U+02048 ⁈ GC=Po SC=Common QUESTION EXCLAMATION MARK
U+02049 ⁉ GC=Po SC=Common EXCLAMATION QUESTION MARK
U+02E2E ⸮ GC=Po SC=Common REVERSED QUESTION MARK
U+03002 。 GC=Po SC=Common IDEOGRAPHIC FULL STOP
U+0A4FF ꓿ GC=Po SC=Lisu LISU PUNCTUATION FULL STOP
U+0A60E ꘎ GC=Po SC=Vai VAI FULL STOP
U+0A60F ꘏ GC=Po SC=Vai VAI QUESTION MARK
U+0A6F3 ꛳ GC=Po SC=Bamum BAMUM FULL STOP
U+0A6F7 ꛷ GC=Po SC=Bamum BAMUM QUESTION MARK
U+0A876 ꡶ GC=Po SC=Phags_Pa PHAGS-PA MARK SHAD
U+0A877 ꡷ GC=Po SC=Phags_Pa PHAGS-PA MARK DOUBLE SHAD
U+0A8CE ꣎ GC=Po SC=Saurashtra SAURASHTRA DANDA
U+0A8CF ꣏ GC=Po SC=Saurashtra SAURASHTRA DOUBLE DANDA
U+0A92F ꤯ GC=Po SC=Kayah_Li KAYAH LI SIGN SHYA
U+0A9C8 ꧈ GC=Po SC=Javanese JAVANESE PADA LINGSA
U+0A9C9 ꧉ GC=Po SC=Javanese JAVANESE PADA LUNGSI
U+0AA5D ꩝ GC=Po SC=Cham CHAM PUNCTUATION DANDA
U+0AA5E ꩞ GC=Po SC=Cham CHAM PUNCTUATION DOUBLE DANDA
U+0AA5F ꩟ GC=Po SC=Cham CHAM PUNCTUATION TRIPLE DANDA
U+0AAF0 ꫰ GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHAN
U+0AAF1 ꫱ GC=Po SC=Meetei_Mayek MEETEI MAYEK AHANG KHUDAM
U+0ABEB ꯫ GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHEI
U+0FE52 ﹒ GC=Po SC=Common SMALL FULL STOP
U+0FE56 ﹖ GC=Po SC=Common SMALL QUESTION MARK
U+0FE57 ﹗ GC=Po SC=Common SMALL EXCLAMATION MARK
U+0FF01 ！ GC=Po SC=Common FULLWIDTH EXCLAMATION MARK
U+0FF0E ． GC=Po SC=Common FULLWIDTH FULL STOP
U+0FF1F ？ GC=Po SC=Common FULLWIDTH QUESTION MARK
U+0FF61 ｡ GC=Po SC=Common HALFWIDTH IDEOGRAPHIC FULL STOP
U+11047 𑁇 GC=Po SC=Brahmi BRAHMI DANDA
U+11048 𑁈 GC=Po SC=Brahmi BRAHMI DOUBLE DANDA
U+110BE 𑂾 GC=Po SC=Kaithi KAITHI SECTION MARK
U+110BF 𑂿 GC=Po SC=Kaithi KAITHI DOUBLE SECTION MARK
U+110C0 𑃀 GC=Po SC=Kaithi KAITHI DANDA
U+110C1 𑃁 GC=Po SC=Kaithi KAITHI DOUBLE DANDA
U+11141 𑅁 GC=Po SC=Chakma CHAKMA DANDA
U+11142 𑅂 GC=Po SC=Chakma CHAKMA DOUBLE DANDA
U+11143 𑅃 GC=Po SC=Chakma CHAKMA QUESTION MARK
U+111C5 𑇅 GC=Po SC=Sharada SHARADA DANDA
U+111C6 𑇆 GC=Po SC=Sharada SHARADA DOUBLE DANDA
To go the other way around — that is, too find properties given a code point instead of finding code points given a set of properties — use the companion uniprops script, which pulls out all the properties of a given code point:
$ uniprops -a . \? \!
U+002E ‹.› \N{FULL STOP}
\pP \p{Po}
All Any ASCII Assigned Basic_Latin Case_Ignorable CI Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn
Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print
X_POSIX_Punct
Age=1.1 Block=Basic_Latin Bidi_Class=Common_Separator BC=CS Bidi_Class=CS Block=ASCII BLK=ASCII Canonical_Combining_Class=0
Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
Line_Break=Infix_Numeric LB=IS Line_Break=IS Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=AT Sentence_Break=ATerm SB=AT
Word_Break=MB Word_Break=MidNumLet WB=MB _Case_Ignorable _X_Begin
U+003F ‹?› \N{QUESTION MARK}
\pP \p{Po}
All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
Word_Break=Other WB=XX Word_Break=XX _X_Begin
U+0021 ‹!› \N{EXCLAMATION MARK}
\pP \p{Po}
All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
Word_Break=Other WB=XX Word_Break=XX _X_Begin
I suspect you should check more into the whole sentence-break properties.
There’s also a 3rd script in the suite, uninames, which does things like this:
$ uninames sentence
; 037E GREEK QUESTION MARK
= erotimatiko
* sentence-final punctuation
* 003B is the preferred character
x (question mark - 003F)
: 003B semicolon
⁚ 205A TWO DOT PUNCTUATION
* historically used to indicate the end of a sentence or change of speaker
* extends from baseline to cap height
x (presentation form for vertical two dot leader - FE30)
x (greek acrophonic epidaurean two - 1015B)
𑂾 110BE KAITHI SECTION MARK
* marks end of sentence
I find those three programs indispensable for exploring Unicode properties. You can install them all using the CPAN Unicode::Tussle suite, or inspect them indivually here.
回答3:
Chinese, Japanese and Korean use 。. Thai uses a space. See this list of Unicode full stop equivalents.
来源：https://stackoverflow.com/questions/9506869/are-there-character-collections-for-all-international-full-stop-punctuations