Does std::wstring support UTF-16 and UTF-32 on Windows?

Question

I'm learning about Unicode and have a few questions that I'm hoping to get answered.

1) I've read that on Linux, a std::wstring is 4-bytes, while on Windows, it's 2-bytes. Does this mean that Linux internal support is UTF-32 while Windows it is UTF-16?

2) Is the use of std::wstring very similar to the std::string interface?

3) Does VC++ offer support for using a 4-byte std::wstring?

4) Do you have to change compiler options if you use std::wstring?

As a sidenote, I came across a string library for working with UTF-8 which has a very similar interface to std::string which provides familiar functionality such as length, substr, find, upper/lower case conversion etc. The library is Glib::ustring.

Please feel free to add any comments or additional advice, because I really need it.

Thank you!

Answer 1

1) wstring is a basic_string<wchar_t> and the size of wchar_t is implementation dependent and encoding agnostic (the standard just says that "its values can represent distinct codes for all members of the largest extended character set specified among the supported locales". But yes, an implementation that has sizeof(wchar_t)=4 bytes supports UTF-32, and sizeof(wchar_t)=2 bytes supports UTF-16.

2) wstring is a basic_string<wchar_t> whereas string is a basic_string<char>, so yes, it is a very similar interface. You will have to use wcout, wcin and wfstream though, and have some other constraints like this.

3) No, MSVC defines wchar_t as unsigned short, which defines and limits wstring as you said. MSVC gives possibility of handling wchar_t as a typedef instead of an internal type. You could imagine then to redefine the typedef, but I suspect this is extreamly risky and evil.

4) No, it's up to you to choose to the string type you want.

5) UTF-32 and the standard : Interestingly, in the very encoding agnostic C++ standard, UTF-32 is mentionned explicitely only for codecvt: "the specialization codecvt <char32_t, char, mbstate_t> converts between the UTF-32 and UTF-8 encoding forms. codecvt converts between the native character sets for narrow and wide characters." This suggests that char32_t would be the portable approach to UTF-32. Unfortunately MSVC doesn't support this type yet.

Answer 2

1) I've read that on Linux, a std::wstring is 4-bytes, while on Windows, it's 2-bytes. Does this mean that Linux internal support is UTF-32 while Windows it is UTF-16?

It is actually wchar_t, not std::wstring, that is 4 bytes on Linux and 2 bytes on Windows. std::wstring is a typedef for std::basic_string<wchar_t>, so std::wstring supports UTF-32 on Linux and UTF-16 on Windows, yes.

2) Is the use of std::wstring very similar to the std::string interface?

Both std::wstring and std::string are typedefs of std:basic_string, so they have the same interface, just different value_type types (wchar_t vs char, respectively).

3) Does VC++ offer support for using a 4-byte std::wstring?

Not for std::wstring itself, no. But you can create your own std::basic_string typedef, eg:

typedef std::basic_string<int32_t> u32string;

In fact, this is exactly how the new C++11 std::u16string and std::u32string types are defined:

typedef std::basic_string<char16_t> u16string;
typedef std::basic_string<char32_t> u32string;

It is also not unheard of to make a typedef of std::basic_string for TCHAR:

typedef std::basic_string<TCHAR> tstring;

As a sidenote, I came across a string library for working with UTF-8 which has a very similar interface to std::string which provides familiar functionality such as length, substr, find, upper/lower case conversion etc. The library is Glib::ustring.

Technically speaking, you can (and many people do) use a standard std::string for UTF-8. Glib::ustring just takes it further by using gunichar (a typedef for guint32) instead of char, and exposes its interfaces to operate in terms of raw Unicode codepoints instead of encoded codeunits.