utf8 <-> utf16: codecvt poor performance

Question

I\'m looking onto some of my old (and exclusively win32 oriented) stuff and thinking about making it more modern/portable - i.e. reimplementing some widely reusable parts in C++

Answer 1

In my own testing, I found that the constructor call for wstring_convert has a massive overhead, at least on Windows. As other answers suggest, you'll probably struggle to beat the native Windows implementation, but try modifying your code to construct the converter outside of the loop. I expect you'll see an improvement of between 5x and 20x, particularly in a debug build.

Answer 2

Win32's UTF8 transcode since Vista uses SSE internally to great effect, something very few other UTF transcoders do. I suspect it will be impossible to beat with even the most highly optimized portable code.

However, this number you've given for codecvt is simply exceptionally slow if it's taking over 10x the time, and suggests a naive implementation. While writing my own UTF-8 decoder, I was able to reach within 2-3x the perf of Win32. There's a lot of room for improvement here, but you'd need to custom implement a codecvt to get it.