Converting String to Android JSONObject loses utf-8

Question

I am trying to get a (JSON formatted) String from a URL and consume it as a Json object. I lose UTF-8 encoding when I convert the String to JSONObject.

This is The f

Answer 1

You can update your code as the following:

    private static String getUrlContents(String theUrl) {
        StringBuilder content = new StringBuilder();
        try {
            URL url = new URL(theUrl);
            URLConnection urlConnection = url.openConnection();
            BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "utf-8"));

            String line;
            while ((line = bufferedReader.readLine()) != null) {
                content.append(line).append("\n");
            }
            bufferedReader.close();
        } catch(Exception e) {
            e.printStackTrace();
        }

        return content.toString().trim();
    }

Answer 2

You've got two encoding issues:

1. The server sends text encoded in a character set. When you setup your InputStreamReader, you need to pass the encoding the server used so it can be decoded properly. The character encoding is usually given in the Content-type HTTP response, in the charset field. JSON is typically UTF-8 encoded, but can also be legally UTF-16 and UTF-32, so you need to check. Without a specified encoding, your system environment will be used when marshalling bytes to Strings, and vice versa . Basically, you should always specify the charset.

2. String messages = new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8"); is obviously going to cause issues (if you have non-ascii characters) - it's encoding the string to ISO-8995-1 and then trying to decode it as UTF-8.

A simple regex pattern can be used to extract the charset value from the Content-type header before reading the inputstream. I've also included a neat InputStream -> String converter.

private static String getUrlContents(String theUrl) {

    try {
        URL url = new URL(theUrl);
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();

        // Get charset field from Content-Type header
        String contentType = urlConnection.getContentType();
        // matches value in key / value pair
        Pattern encodingPattern = Pattern.compile(".*charset\\s*=\\s*([\\w-]+).*");
        Matcher encodingMatcher = encodingPattern.matcher(contentType);
        // set charsetString to match value if charset is given, else default to UTF-8
        String charsetString = encodingMatcher.matches() ? encodingMatcher.group(1) : "UTF-8";

        // Quick way to read from InputStream.
        // \A is a boundary match for beginning of the input
        return new Scanner(is, charsetString).useDelimiter("\\A").next();
    } catch(Exception e) {
        e.printStackTrace();
    }

    return null;
}

Answer 3

You're telling Java to convert the string (with key message) to bytes using ISO-8859-1 and than to create a new String from these bytes, interpreted as UTF-8.

new String(reader.getString("messages").getBytes("ISO-8859-1"), "UTF-8");

You could simply use:

String messages = reader.getString("messages");

Answer 4

Not sure if this will help, but you might be able to do something like this:

JSONObject result = null;
String str = null;
try 
{           
    str = new String(output, "UTF-8");
    result = (JSONObject) new JSONTokener(str).nextValue();
} 
catch (Exception e) {}

String messages = result.getString("messages");