Populating SELECT with large JSON data set via ColdFusion (Lucee) very slow

问题

Please forgive me if I have provided more information than required for this question. :D

I am building an application that pulls large JSON data-sets from a remote machine. However, I am working within a secure environment that separates application servers with firewalls, etc. Because of this I have had to do a bit of fudging (using SSH) to get the data I need. I have requested additional ports to be opened so I could bypass using SSH but was denied.

Here is the physical path to get my data over SSH: My Application Server -> Secure Server with Curl -> Multiple unique application servers with JSON data.

To abide by the security rules; I wrote the CFML code below on my application server. (server is running Linux, Lucee 4.5, MySQL and Apache)

<cfset secure_server = "secure_server.my.example.com">
<cftry>
    <cfexecute 
    name="/usr/bin/ssh" 
    arguments="#secure_server# ./RemoteDirectory/server_list.sh" 
    timeout="10" 
    variable="server_list">
    </cfexecute>
    <cfcatch type="any">
        <cfoutput>
            #cfcatch.message#<br>#cfcatch.detail#<br>
        </cfoutput>
    </cfcatch>
</cftry>

The above code simply SSHs to the secure server and runs a script called server_list.sh. The important bit of the script is below:

curl -s -k -b cookie.jar -c cookie.jar https://unique_application_server/Path/To/Server/list -H "Accept:application/json"

That curl script returns a JSON string that looks similar (but with far more data) to this:

{
  "os" : "Linux ",
  "fqdn" : "server1.my.example.com",
  "disk_status" : "OK",
  "system_model" : "HP",
  "type" : "Server"
},
{
  "os" : "Linux ",
  "fqdn" : "server2.my.example.com",
  "disk_status" : "OK",
  "system_model" : "Dell",
  "type" : "Server"
},

The amount of JSON data sent, at this time, is about 3MB. However, that data will grow exponentially over time.

Once I get the JSON back to my application server; I use CF's deserializeJSON function then grab just the field I need. In this case its the FQDN field. Since the data from the JSON is a full domain, I needed to chomp just the hostname. The "new_label" code takes care of that. There is probably a better REGEX or a better method than the one I am using, but this works.

<cfset arrayOfStructs = deserializeJson(server_list)>
<select multiple="multiple" class="searchable" name="searchable[]">
    <cfloop array="#arrayOfStructs#" index="PARENT">
        <cfset new_label = ReReplace(PARENT.fqdn, "(.[^.]*)$", "")>
        <cfset new_label = ReReplace(new_label, "(.[^.]*)$", "")>
        <cfset new_label = ReReplace(new_label, "(.[^.]*)$", "")>
        <cfoutput><option value="#PARENT.fqdn#">#new_label#</option></cfoutput>
    </cfloop>
</select>

The above code populates the field with the hostnames of servers from the JSON. The OPTION field would look something like this:

<option value="server1.my.example.com">server1</option>

A couple things of note that may (or may not) help. I am using jQuery and a multi-select.js that turns the SELECT into a side-by-side field. The multi-slect is from here: http://loudev.com/

Second, all servers in the path are fairly powerful and are all connected over 10gb. So besides the firewalls between, there should not be any bottle necks.

No matter the browser I use, the code above works, but is rather slow when it populates the hostname field. Because the JSON data I am pulling updates about every minute, I am not able to store it long term in the database on my application server.

Knowing all that, my question is: How could I speed up the form field so the user does not have to wait for the page to load 2500+ servernames?

Could I leverage jQuery or the MySQL database to speed things up without running into stale data? Could it simply be my method of getting the hostname from the FQDN? Should I scrap this method and try something else? Have I overlooked something here on Stack that has my answer? (I spent a lot of time searching Q/A here but found nothing. Perhaps I was not looking hard enough.)

Any other information you nice folks need? Thanks! :D

UPDATE --

@Alex. I do that very test now. Code is below. Good suggestion though!

<cfset tickBegin = GetTickCount()>
<!--- THE CODE THAT RUNS THE JSON QUERY ABOVE --->  
<cfset tickEnd = GetTickCount()>
<cfset testTime = tickEnd - tickBegin>
<cfoutput>Page(#testTime#ms)</cfoutput>

As far as caching the data, that is part of my question. How to cache? What method to use? How can I cache the JSON without displaying stale data? Is this an instance where AJAX would be useful?

UPDATE 2 --

Could I be going about this the wrong way? I know that sometimes you will need to display large amounts of data back to your users. Would simply adding a BlockUI and a progress bar stating please wait be an OK solution? I would prefer to figure out a better solution honestly but I am open to suggestions for sure! Thoughts?

来源：https://stackoverflow.com/questions/37733573/populating-select-with-large-json-data-set-via-coldfusion-lucee-very-slow

标签

jquery

json

coldfusion

lucee

large-data