Stumped with Unicode, Boost, C++, codecvts

Question

In C++, I want to use Unicode to do things. So after falling down the rabbit hole of Unicode, I\'ve managed to end up in a train wreck of confusion, headaches and locales.

Answer 1

Okay, after a long few months I've figured it out, and I'd like to help people in the future.

First of all, the codecvt thing was the wrong way of doing it. Boost.Locale provides a simple way of converting between character sets in its boost::locale::conv namespace. Here's one example (there's others not based on locales).

#include 
namespace loc = boost::locale;

int main(void)
{
  loc::generator gen;
  std::locale blah = gen.generate("en_US.utf-32");

  std::string UTF8String = "Tésting!";
  // from_utf will also work with wide strings as it uses the character size
  // to detect the encoding.
  std::string converted = loc::conv::from_utf(UTF8String, blah);

  // Outputs a UTF-32 string.
  std::cout << converted << std::endl;

  return 0;
}

As you can see, if you replace the "en_US.utf-32" with "" it'll output in the user's locale.

I still don't know how to make std::cout do this all the time, but the translate() function of Boost.Locale outputs in the user's locale.

As for the filesystem using UTF-8 strings cross platform, it seems that that's possible, here's a link to how to do it.