Loading large amount of data into memory - most efficient way to do this?

Question

I have a web-based documentation searching/viewing system that I\'m developing for a client. Part of this system is a search system that allows the client to search for a t

Answer 1

Instead of using $.getScript to load JavaScript files containing function calls, consider using $.getJSON. This may boost performance. The files would now look like this:

{
    "key" : 0,
    "values" : [0,1,2,3,4,5,6,7,8]
}

After receiving the JSON response, you could then call AddToBookData on it, like this:

function AddToBookData(json) {
     BookData[BookIndex].push([json.key,json.values]);
}

If your files have multiple sets of calls to AddToBookData, you could structure them like this:

[
    {
        "key" : 0,
        "values" : [0,1,2,3,4,5,6,7,8]
    },
    {
        "key" : 1,
        "values" : [0,1,2,3,4,5,6,7,8]
    },
    {
        "key" : 2,
        "values" : [0,1,2,3,4,5,6,7,8]
    }
]

And then change the AddToBookData function to compensate for the new structure:

function AddToBookData(json) {
    $.each(json, function(index, data) {
        BookData[BookIndex].push([data.key,data.values]);
    });
}  

Addendum
I suspect that regardless what method you use to transport the data from the files to the BookData array, the true bottleneck is in the sheer number of requests. Must the files be fragmented into 40-100? If you change to JSON format, you could load a single file that looks like this:

{
    "file1" : [
        {
            "key" : 0,
            "values" : [0,1,2,3,4,5,6,7,8]
        },
        // all the rest...
    ],
    "file2" : [
        {
            "key" : 1,
            "values" : [0,1,2,3,4,5,6,7,8]
        },
        // yadda yadda
    ]
}

Then you could do one request, load all the data you need, and move on... Although the browser may initially lock up (although, maybe not), it would probably be MUCH faster this way.

Here is a nice JSON tutorial, if you're not familiar: http://www.webmonkey.com/2010/02/get_started_with_json/