Why does anyone use an encoding other than UTF-8? [closed]

Answer 1

Wikipedia lists advantages and disadvantages of UTF-8 as compared to a variety of other encodings:

http://en.wikipedia.org/wiki/UTF-8#Advantages_and_disadvantages

The most important disadvantages are IMHO that UTF-8 might use significantly more space especially in Asian languages such as Chinese, Japanese or Hindi and that not all code points have the same size which makes measurements more difficult and many string operations such as search inefficient.

Answer 2

The reasons for using non-Unicode 8-bit character sets / encodings are all back compatibility of some kind, and/or inertia. For that matter, the most frequent reasons for using UTF-8 are compatibility with standards like XML that mandate or prefer UTF-8.

Differences in the number of bytes you think text will take up in different encodings, especially in storage, are mostly theoretical. In real world situations, compatibility requirements are more important. If compression is used, the size differences go away anyway. Even if compression is not used, total text size is hard to predict and is rarely a deciding factor.

When converting legacy code that used non-Unicode 8-bit encodings, using UTF-16 can be a tool for making sure all code has been converted, because mismatches can be caught as compile-time type errors. Many languages, runtimes and libraries like Javascript, JVM, .NET, ICU use 16-bit strings and UTF-16, even though storage and Internet protocols are usually 8-bit.

Answer 3

Imagine all files to consider are in GB2312 (China mainland standard). Then you might choose GB18030 as Unicode encoding instead. They are compatible the same way as all ASCII is UTF-8. That is useful in China mainland!

You might decide even quicker when you find out that both mentioned GB-standards are required in your IT-product by law (as far as I have heard), if you want to ship in China (mainland).

Another upside is that GB2312, and as such GB18030 as well, are also ASCII compatible.

It is algorithmically not so robust, though. – So if you have no political reasons or any GB2312 legacy, it makes no sense to use it. But if you do, here you got your answer.

Answer 4

At my previous employer we used iso-8859-1 for some of our ASP pages to match the collation of our SQL Server, which as you can guess was not Unicode. I wanted to change the collation, but the manager said to wait till we upgrade our SQL Server to do it. Needless to say it never happened - I haven't been with them for a little over a year now, so I don't know if they finally did it.

Answer 5

UTF-8 is very efficient at encoding plain English text (same as ASCII). If your user base is likely to be mostly, say, Chinese, you will be much better off using UTF-16.

For more information, see The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets.

Answer 6

Because they do not know better. The only valid criticism to utf-8 is that encodings for common Asian languages are oversized from other encodings. UTF-8 is superior because

• It is ASCII compatible. Most known and tried string operations do not need adaptation.
• It is Unicode. Anything that isn't Unicode shouldn't even be considered in this day and age. If you have important data in encoding X, spend two minutes on Google and write a conversion function. Even if you have to interface with sourceless legacy app Z, you can run your communications through a pipe so that your logic stays in the 21st century.
• UTF-16 isn't fixed length either and assuming it is like many do, will only cause terrible bugs.
• Additionally Unicode is very complex and it is almost certain than any fixed-size algorithm adapted from ASCII will yield bad results even in UTF-32.

Say you have this UTF-16 string.

[0][1][2][F|3] [4] [5]

And you want to insert a character with code 8 between [3] and [4] you would do insert(5,8)

If you don't check for characters outside BMP(serially as in UTF-8 as you cannot know how many double sized characters you have) you get:

[0][1][2][F|8][3][4][5]

Two new garbage characters. So much for your fixed size encoding. You can of course disallow such characters altogether, but then when your code interfaces with the real world, you might find your program saves the profile for this user who lives in rm -Rf / in .profile instead of [Classical Chinese Proverb].profile.

Or just an angry user that cannot write his thesis on Classical Chinese Proverbs with your software.