Utf-8 in c++: quick & dirty tricks

Question

I am aware that there are been various questions about utf-8, mainly about libraries to manipulate utf-8 \'string\' like objects.

However, I am working on an \'internati

Answer 1

Well this dirty trick will not work. First, what is the value of mask after this:

   const unsigned char mask = 0x11000000;
   const unsigned char notUtf8Begin = 0x10000000;

Perhaps you are mixing hex representation with binary.

Second, as you correctly say in utf-8 encoding, a character may be several bytes long. std::count_if will iterate through all bytes in a UTF8 sequence. But what you actually need is to look at leading byte for every character and skip the rest until the next character comes.

It will not be hard to implement a single cycle which does the calculation and jumping forward using the simple mask table for leading bytes.

At the end you get the same O(n) for checking the characters and it will work with every UTF8 string.

Answer 2

We handle it also like this in OpenLieroX (which is really fine in a game I think).

We have a bunch of useful functions/algorithms for such UTF-8 std::strings. See Unicode.h and Unicode.cpp. For example, there are UTF8 iterators, some simple manipulation operators (insert or erase), upper/lower case conversions, case independent search, etc.

But don't expect those functions to be always correct. For example, they don't know really about combining diacritics or possible different ways to encode the same text.

Answer 3

Sorting UTF_8 as binary will not sort in 'Unicode' order. BOCU-1 would. As was said, your "as expected" is a pretty low bar for non-English content.