Converting String to Android JSONObject loses utf-8

Question

I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject.

This is The f

Answer 1

You've got two encoding issues:

1. The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.

2. String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.

A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.

private static String getUrlContents(String theUrl) {

    try {
        URL url = new URL(theUrl);
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();

        // Get charset field from Content-Type header
        String contentType = urlConnection.getContentType();
        // matches value in key / value pair
        Pattern encodingPattern = Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
        Matcher encodingMatcher = encodingPattern.matcher(contentType);
        // set charsetString to match value if charset is given, else default to UTF-8
        String charsetString = encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";

        // Quick way to read from InputStream.
        // \A is a boundary match for beginning of the input
        return new Scanner(is, charsetString).useDelimiter("\\A").next();
    } catch(Exception e) {
        e.printStackTrace();
    }

    return null;
}