How to convert UTF-8 encoded std::string to UTF-16 std::string

Question

How can i convert UTF-8 encoded std::string to UTF-16 std::string? Is it possible?

And no, i can\'t use std::wstring in my case.

Windows, MSVC-11.0.

Answer 1

How about trying like this:-

std::string s = u8"Your string";

// #include <codecvt>
std::wstring_convert<std::codecvt<char16_t,char,std::mbstate_t>,char16_t> convert;

std::u16string u16 = convert.from_bytes(s);
std::string u8 = convert.to_bytes(u16);

Also check this for UTF to UTF conversion.

From the docs:-

The specialization codecvt converts between the UTF-16 and UTF-8 encoding schemes, and the specialization codecvt converts between the UTF-32 and UTF-8 encoding schemes.

Answer 2

I've come across dozens of such problems trying to do this and similar with Visual Studio, and just gave up. There is a known issue linking when doing conversions with e.g. std::wstring's convert and using std::codecvt.

Please see here: Convert C++ std::string to UTF-16-LE encoded string

What I did to resolve my problem was copied in the code from a kind poster, which uses the iconv library. Then all I had to do was call convert(my_str, strlen(my_str), &used_bytes), where my_str was a char[], strlen(my_str) was its length, and size_t used_bytes = strlen(my_str)*3; I just gave it enough bytes to work with. In that function, you can change iconv_t foo = iconv_open("UTF-16", "UTF-8"), investigate the setlocale() and creation of the enc string passed to iconv_open() above in the function which is sitting there in all it's glory in the link above.

The gotcha is compiling and using iconv, it almost expects Cygwin or such on Windows, but you can use that with Visual Studio. There is a purely Win32 libiconv at https://github.com/win-iconv/win-iconv which might suit your needs.

I would say give iconv a try, and see how it goes in a short test program. Good luck!