Perl's JSON::XS not encoding UTF8 correctly?

Question

This simple code segment shows an issue I am having with JSON::XS encoding in Perl:

#!/usr/bin/perl
use strict;
use warnings;
use JSON::XS; 
use utf8;
binmode ST         


        

Answer 1

encode_json (short for JSON::XS->new->utf8->encode) encodes using UTF-8, then you are re-encoding it by printing it to STDOUT to which you've added an encoding layer. Effectively, you are doing encode_utf8(encode_utf8($uncoded_json)).

Solution 1

use open ':std', ':encoding(utf8)';  # Defaults
binmode STDOUT;                      # Override defaults
print encode_json(\%data);

Solution 2

use open ':std', ':encoding(utf8)';    # Defaults
print JSON::XS->new->encode(\%data);   # Or to_json from JSON.pm

Solution 3

The following works with any encoding on STDOUT by using \u escapes for non-ASCII:

print JSON::XS->new->ascii->encode(\%data);

---

In the comments, you mention it's actually a CGI script.

#!/usr/bin/perl
use strict;
use warnings;

use utf8;                      # Encoding of source code.
use open ':encoding(UTF-8)';   # Default encoding of file handles.
BEGIN {
   binmode STDIN;                       # Usually does nothing on non-Windows.
   binmode STDOUT;                      # Usually does nothing on non-Windows.
   binmode STDERR, ':encoding(UTF-8)';  # For text sent to the log file.
}

use CGI      qw( -utf8 );
use JSON::XS qw( ); 

{
   my $cgi = CGI->new();
   my $data = { code => "Gewürztraminer" };
   print $cgi->header('application/json');
   print encode_json($data);
}