How to deal with a large JSON object on Android

Question

I\'m making an Android app that runs a ASP.NET WebService. Webservice sends a JSON object and app parses the object and displays on the screen. In one case, JSON object is t

Answer 1

If data is large then try to save it in the database, then deal with it using SQLite. (but not recommended if its dynamic)

To parse json object use gson or jackson. This will help reduce the memory consumption significantly as the json data being parsed partially. get Gson, jackson here https://sites.google.com/site/gson/gson-user-guide http://jackson.codehaus.org/

A jackson example http://www.mkyong.com/java/jackson-streaming-api-to-read-and-write-json/

Answer 2

The following class ApiUrlClass.java has all methods you require. Please read the comments of the class which I wrote. That will help you to do what you require. This also utilises transparent.

import android.graphics.Bitmap;
import android.net.Uri;
import android.os.Build;
import android.util.Log;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.content.ByteArrayBody;
import org.apache.http.entity.mime.content.ContentBody;
import org.apache.http.entity.mime.content.StringBody;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.ByteArrayOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.SocketTimeoutException;
import java.net.URL;
import java.net.UnknownHostException;

import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLPeerUnverifiedException;

/*
Usage of the class

Create all the necessary API Call methods you need.
And either use a Thread or AsyncTask to call the following.

    JSONObject response = ApiUrlCalls.login("username", "passowrd");

After the response is obtained, check for status code like

    if(response.getInt("status_code") == 200){
        //TODO: code something
    } else {
        //TODO: code something
    }
*/

public class ApiUrlCalls {

    private String HOST = "https://domain/path/"; //This will be concated with the function needed. Ref:1

    /*
        Now utilizing the method is so simple. Lets consider a login function, which sends username and password.
        See below for example.
    */

    public static JSONObject login(String username, String password){

        String functionCall = "login";
        Uri.Builder builder = new Uri.Builder()
                .appendQueryParameter("username", username)
                .appendQueryParameter("password", password);

        /*
            The return calls the apiPost method for processing.
            Make sure this should't happen in the UI thread, orelse, NetworkOnMainThread exception will be thrown.
        */
        return apiPost(builder, functionCall);

    }

    /*
        This method is the one which performs POST operation. If you need GET, just change it
        in like Connection.setRequestMethod("GET")
    */
    private static JSONObject apiPost(Uri.Builder builder, String function){
        try {
            int TIMEOUT = 15000;
            JSONObject jsonObject = new JSONObject();
            try {
                URL url = null;
                String response = "";

                /*
                    Ref:1
                    As mentioned, here below, in case the function is "login",
                    url looks like https://domain/path/login

                    This is generally a rewrited form by .htaccess in server.
                    If you need knowledge on RESTful API in PHP, refer 
                    http://stackoverflow.com/questions/34997738/creating-restful-api-what-kind-of-headers-should-be-put-out-before-the-response/35000332#35000332

                    I have answered how to create a RESTful API. It matches the above URL format, it also includes the .htaccess
                */

                url = new URL(HOST + function);

                HttpsURLConnection conn = null;
                conn = (HttpsURLConnection) url.openConnection();
                assert conn != null;
                conn.setReadTimeout(TIMEOUT);
                conn.setConnectTimeout(TIMEOUT);
                conn.setRequestMethod("POST");
                conn.setDoInput(true);
                conn.setDoOutput(true);

                String query = builder.build().getEncodedQuery();

                OutputStream os = conn.getOutputStream();
                BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(os, "UTF-8"));
                writer.write(query);
                writer.flush();
                writer.close();
                os.close();
                conn.connect();


                int responseCode = conn.getResponseCode();
                String responseMessage = conn.getResponseMessage();
                jsonObject.put("status_code", responseCode);
                jsonObject.put("status_message", responseMessage);

                /*The if condition below will check if status code is greater than 400 and sets error status
                even before trying to read content, because HttpUrlConnection classes will throw exceptions
                for status codes 4xx and 5xx. You cannot read content for status codes 4xx and 5xx in HttpUrlConnection
                classes. 
                */

                if (jsonObject.getInt("status_code") >= 400) {
                    jsonObject.put("status", "Error");
                    jsonObject.put("msg", "Something is not good. Try again later.");
                    return jsonObject;
                }

                String line;
                BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));

                while ((line = br.readLine()) != null) {
                    response += line;
                }
                //Log.d("RESP", response);

                /*
                    After the actual payload is read as a string, it is time to change it into JSON.
                    Simply when it starts with "[" it should be a JSON array and when it starts with "{"
                    it is a JSONObject. That is what hapenning below.
                */
                if(response.startsWith("[")) {
                    jsonObject.put("content", new JSONArray(response));
                }
                if(response.startsWith("{")){
                    jsonObject.put("content", new JSONObject(response));
                }


            } catch(UnknownHostException e) {
            //No explanation needed :)
                jsonObject.put("status", "UnknownHostException");
                jsonObject.put("msg", "Check your internet connection");
            } catch (SocketTimeoutException){
            //This is when the connection timeouts. Timeouts can be modified by TIMEOUT variable above.
                jsonObject.put("status", "Timeout");
                jsonObject.put("msg", "Check your internet connection");
            } catch (SSLPeerUnverifiedException se) {
            //When an untrusted SSL Certificate is received, this happens. (Only for https.)
                jsonObject.put("status", "SSLException");
                jsonObject.put("msg", "Unable to establish secure connection.");
                se.printStackTrace();
            } catch (IOException e) {
            //This generally happens when there is a trouble in connection
                jsonObject.put("status", "IOException");
                jsonObject.put("msg", "Check your internet connection");
                e.printStackTrace();
            } catch(FileNotFoundException e){ 
            //There is no chance that this catch block will execute as we already checked for 4xx errors
                jsonObject.put("status", "FileNotFoundException");
                jsonObject.put("msg", "Some 4xx Error");
                e.printStackTrace();
            } catch (JSONException e){ 
            //This happens when there is a troble reading the content, or some notice or warnings in content, 
            //which generally happens while we modify the server side files. Read the "msg", and it is clear now :)
                jsonObject.put("status", "JSONException");
                jsonObject.put("msg", "We are experiencing a glitch, try back in sometime.");
                e.printStackTrace();
            } return jsonObject;

        } catch (JSONException e) {
            e.printStackTrace();
        }
        return null;
    }

}

Answer 3

Preferably try to break the Json Object to smaller object and get from webService ,

or get data in parts and if u cant do that

You have to use a streaming JSON parser. For Android u can use these 2:

1. GSON

2. Jackson

GSON Streaming is explained at: https://sites.google.com/site/gson/streaming

I personally like Gson .

Answer 4

Sending the large size data from server to mobile. JSON is light weight. If you want to pass the data using more efficient way then passes it in pagination. If you want to use more lighter protocol than JSON then implement the below google protocol which are really useful, which are supporting major languages. Below are smaller Serialised data structure. Google's data interchange protocol.

1.Google Protocol

2.Flat Buffers

3.Nano-proto buffers

Hope this will be useful you.

Answer 5

You could embed your JSON in your app's code as you suggested, but this will be a bad approach if the JSON is dynamic. Then you would need to push an update for your app whenever the JSON changes.

A better solution would be to paginate the JSON that you generate from your WebService, i.e., break the JSON into smaller parts that you can fetch sequentially in separate API calls.

Answer 6

First thing: If there is a crash or exception in your code, you'll probably want to post that. "Failed Binder Exception" is a bit too vague to understand what you're doing.

If you really want to ship your Android app with JSON embeffffd inside it (to avoid having to fetch it from a server, consider storing it as an asset and access it using AssetManager. You basically drop the file with the json in your app's assets folder and read them out with AssetManager.

If you still want to download it from the server and act on it, consider using streaming APIs to download and parse the JSON. Android's JSONObject does not do this and it insists on having the entire JSON string in memory before it can be parsed.

If you want to stream directly from a URL download into a streaming parser (such as GSON), try something along these lines. First get an InputStream from the URL you're trying to fetch:

    URL u = new URL(url);
    URLConnection conn = u.openConnection();
    InputStream is = new BufferedInputStream(conn.getInputStream());

Then feed that InputStream directly to your streaming parser. This should prevent the need to pull the entire response into memory before parsing, but you'll still need enough memory to contain all the objects that the parser creates:

    GsonBuilder gb = new GsonBuilder(); // configure this as necessary
    Gson gson = gb.create();
    final Result response = gson.fromJson(
        new InputStreamReader(is, Charset.forName("UTF-8")),
        Result.class
    );

"Result" here is a class that will contain the data from the JSON response. You'll have to make sure all the mappings work for your data, so read up on GSON and do whatever works for your case.

You can also use GSON to parse the JSON data if you store it in an asset. Just hand it the InputStream of the asset data and it works the same way.