Perl's JSON::XS not encoding UTF8 correctly?

Question

This simple code segment shows an issue I am having with JSON::XS encoding in Perl:

#!/usr/bin/perl
use strict;
use warnings;
use JSON::XS; 
use utf8;
binmode ST         


        

Answer 1

JSON::XS encodes its output into octets. It means the external representation of encoded utf8 string, but it is not unicode string. For more details see perlunicode. In short, content of $json_text is prepared for transmitting by IO handler in binary code. If you create scalar content of $data{code} after use utf8; you have scalar containing internally encoded unicode characters string. (Which is internally encoded as utf8 but it is implementation detail you should not rely on. Pragma use utf8; means the source code is encoded as utf8 and nothing else.) If you would like to output both scalars in utf8 encoded IO handler you have to transform $json_string into internal unicode chracters string.

use strict;
use warnings;
use JSON::XS; 
use utf8;
binmode STDOUT, ":encoding(utf8)";

my (%data);

$data{code} = "Gewürztraminer";
print "data{code} = " . $data{code} . "\n";

my $json_text = encode_json \%data;
utf8::decode($json_text);
print $json_text . "\n";

Or how it is intended to use, output encoded string using IO handler in binary mode.

my $json_text = encode_json \%data;
binmode STDOUT;
print $json_text . "\n";

Try

print utf8::is_utf8($json_text) ? "UTF8" : "OCTETS" . "\n";

to see what is inside.