OutOfMemoryError when using Gson to parse a large JSON response

Question

URL url = new URL(\"http://pubapi.cryptsy.com/api.php?method=orderdatav2\");
CryptsyCurrencyPairsReturn response = gson.fromJson(new InputStreamReader(url.openStream()),         


        

Answer 1

From my experience, you need to stream the JSON data.
And yes you can use google GSON to stream JSON data this is an example how to do it :

APIModel result = new APIModel();
        try {
            HttpResponse response;
            HttpClient myClient = new DefaultHttpClient();
            HttpPost myConnection = new HttpPost(APIParam.API_001_PRESENT(
                    serial_id, api_key));
            try {
                response = myClient.execute(myConnection);
                Reader streamReader = new InputStreamReader(response
                        .getEntity().getContent());
                JsonReader reader = new JsonReader(streamReader);
                reader.beginObject();
                while (reader.hasNext()) {

                    String name = reader.nextName();

                    if (name.equals("result")) {
                        if (reader.nextString() == "NG") {
                            result.setResult(Util.API_001_RESULT_NG);
                            break;
                        }
                    } else if (name.equals("items")) {
                        result = readItemsArray(reader);
                    } else {
                        reader.skipValue(); // avoid some unhandle events
                    }
                }

                reader.endObject();
                reader.close();
            } catch (Exception e) {
                e.printStackTrace();
                result.setResult(Util.API_001_RESULT_NG);
            }
        } catch (Exception e) {
            e.printStackTrace();
            result.setResult(Util.API_001_RESULT_NG);
        }

readItemsArray function :

// read items array
    private APIModel readItemsArray(JsonReader reader) throws IOException {
        APIModel result = new APIModel();
        String item_name, file_name, data;
        result.setResult(Util.API_001_RESULT_OK);

        reader.beginArray();
        while (reader.hasNext()) {
            item_name = "";
            file_name = "";
            data = "";
            reader.beginObject();
            while (reader.hasNext()) {
                String name = reader.nextName();
                if (name.equals("name")) {
                    item_name = reader.nextString();
                } else if (name.equals("file")) {
                    file_name = reader.nextString();
                } else if (name.equals("data")) {
                    data = reader.nextString();
                } else {
                    reader.skipValue();
                }
            }
            reader.endObject();
            result.populateModel("null", item_name, file_name, data);
        }
        reader.endArray();
        return result;
    }

API Model Class :

public class APIModel {
    private int result;
    private String error_title;
    private String error_message;
    private ArrayList<String> type;
    private ArrayList<String> item_name;
    private ArrayList<String> file_name;
    private ArrayList<String> data;

    public APIModel() {
        result = -1;
        error_title = "";
        error_message = "";
        setType(new ArrayList<String>());
        setItem_name(new ArrayList<String>());
        setFile_name(new ArrayList<String>());
        setData(new ArrayList<String>());
    }

    public void populateModel(String type, String item_name, String file_name, String data) {
        this.type.add(type);
        this.item_name.add(item_name);
        this.file_name.add(file_name);
        this.data.add(data);
    }

    public int getResult() {
        return result;
    }

    public void setResult(int result) {
        this.result = result;
    }

    public String getError_title() {
        return error_title;
    }

    public void setError_title(String error_title) {
        this.error_title = error_title;
    }

    public String getError_message() {
        return error_message;
    }

    public void setError_message(String error_message) {
        this.error_message = error_message;
    }

    public ArrayList<String> getType() {
        return type;
    }

    public void setType(ArrayList<String> type) {
        this.type = type;
    }

    public ArrayList<String> getItem_name() {
        return item_name;
    }

    public void setItem_name(ArrayList<String> item_name) {
        this.item_name = item_name;
    }

    public ArrayList<String> getFile_name() {
        return file_name;
    }

    public void setFile_name(ArrayList<String> file_name) {
        this.file_name = file_name;
    }

    public ArrayList<String> getData() {
        return data;
    }

    public void setData(ArrayList<String> data) {
        this.data = data;
    }


}

before I use the streaming API from google GSON I also got OOM error because the JSON data I got is very big data (many images and sounds in Base64 encoding) but with GSON streaming I can overcome that error because it reads the data per token not all at once. And for Jackson JSON library I think it also have streaming API and how to use it almost same with my implementation with google GSON. I hope my answer can help you and if you have another question about my answer feel free to ask in the comment :)

Answer 2

I'd recommend looking at JsonReader (API 11+), or the other options as recommended here.

Answer 3

Parsing large data in one-go is always tricky and troublesome. However Gson comes with some nice features to support that too.

You should look for com.google.gson.stream.JsonReader to perform json parsing using streams. This will allow you to parse the data in an incremental order while its being downloaded and will spare devices from OutOfMemory errors.

For more info, read this.