Improve highcharts performance for large amounts of data

Question

I am trying to get a larger amount of data. Sample data is below

1850/01   -0.845   -0.922   -0.748   -1.038   -0.652   -1.379   -0.311   -1.053   -0.636   -1.41         


        

Answer 1

my team came across the similar problem but in our case we had 3 nested loops and when it was processing the data it was blocking the main thread which caused bad user experience. so i tackled this problem by deferring the processing data. which did the desired job of not blocking the main thread and did the processing much faster.

you can also use boost module developed by highchart (https://www.highcharts.com/blog/news/175-highcharts-performance-boost/)

Answer 2

new Array(length)

Specify the array's length rather than creating an empty array. Writing to an existing offset in an array is substantially faster than creating a new offset each time you write an item.

var data = new Array(lines.length);  // Specify array length

$.each(lines, function(index, row) {
  var cells = row.split(','),
  series = {
    type: 'line',
    data: new Array(cells.length)  // Specify array length
  };

  $.each(cells, function(itemNo, item) {
    if (itemNo == 0) {
      series.name = item;
    } else {
      series.data.push(parseFloat(item));
    }
  });

  data.push(series);
});

Answer 3

Given the data you are displaying is not updating once a month, I feel like generating the chart for every view is a big waste of resources for your clients.

Indeed, it would be pretty easy for you to generate the chart without changing anything that you are doing now, but then extracting the resulting SVG and serving it simply to your visitors.

For that you simply have to use the getSVG method of HighCharts, that will return the SVG in a stringified form.

I don't really know if you want to have some sort of process to auto update the chart, but you could use a cron job for this purpose. Keep in mind that you would have to do something anyway even with your initial approach to update the data.

---

Regarding your script, first thing to notice is that you are using $.each, which is pretty bad in term of performance comparing to the vanilla equivalents. As demonstrated by this jsPerf, I get 3,649 Op/s with $.each whereas for loops could get you up to 202,755 Op/s which is insanely faster! Since you also doing a double loop, the difference would be ^2.

But again, since the data is not updated often, this part of the script could be removed entirely and converted into a JSON corresponding to the output of the function, that HighCharts could load directly, skipping the entire processing and transformation of your CSV.

---

If using HighCharts is not a requirement, you could use react-vis, which is a collection of React components focused around Data Visualization. It's build as an api on top of SVG, .

I've made a demo that you can checkout on CodePen with the same data as you to plot the monthly temperatures since 1850.

const {
  HorizontalGridLines,
  XAxis,
  XYPlot,
  YAxis,
  LineMarkSeries,
} = reactVis

const color = d3.scaleLinear()
  .domain([1850, 2017])
  .range(['yellow', 'red'])

const Chart = () => (
  <XYPlot width={window.innerWidth - 100} height={window.innerHeight - 100}>
    <XAxis title='Month' />
    <YAxis title='Temperature' />
    <HorizontalGridLines />
    {Object.keys(data).map(key => (
      <LineMarkSeries color={color(key)} data={data[key]} key={key} />
    ))}
  </XYPlot>
)

ReactDOM.render(<Chart />, document.querySelector('#root'))

I also use the scaleLinear method of d3 to see the change over the years since I thought it would be an interesting information to show, but it can be changed depending of your needs.

It's using SVGs, but we are also working on integration with webgl by the intermediary of deck.gl which would allow for even more optimizations gains, still not ready yet and I'm not sure you would really need that much, but worth noting.

_{Disclaimer: I currently work for Uber, which made deck.gl and react-vis.}

Answer 4

I'd use underscore.js.

 var result = _.map(lines, function (row) {
    return { type: 'line', series: _.map(row.data, function (item, index) {
      if (index >= 1)
        return parseFloat(item);
    }), name: row.data[0]}
  });

(I'm writing this from my phone on a plane, so my apologies if there is are any typos or things I'm missing. Just trying to help!)

Answer 5

You don't show what is your choice for the xAxis, but if massaging the data in the server side is not an option what I would suggest is:

1. First do a call to load the data.
2. Show at spinning wheel where the chart is going to be displayed
3. Start parsing the dataset but don't load it all at once but in batches and redraw the chart as you parse the rest of the file.

This should avoid the freezing and give the visual indication that the data is being loaded.

Edit: Response to comment I don't think is any longer an issue loading the data but on how to display the data in a meaningful way. Even if it loads in 2 seconds the result would be thousands of lines overlapping and totally unreadable.

You should default few hand picked values (ex Current month and same month and 1, 5, 10 years ago) and then allow the user to change the selection select up to a reasonable number of years/month (ex Compare July for 1980, 2010 and 2017) then plot those values updating the chart with (chart.series[0].update(....). There were no columns in the dataset so I assumed it was a column per month but I see now that they seem to be one per row and 11 columns represent some other data.

I don't think for a polar chart doing averages or doing rollups of the data is the best way to go but allowing to compare specific years and month with the exact data, whatever those values are :)

Answer 6

Situation:

I think the best and more appropriate solution would be to process and serve chunks of data, it would synchronize the display of points on your Chart and avoid loading the whole data once on document load, even if your data isn't as large as you described it in your post title.

But after reading your question, your comments and other posts, I can see that you are getting all data from one file, and you aren't processing data on server side so server side chunks is not an option for you.

Alternative:

Well if you can't process your data on server side why don't you process it on client side, HighCharts has two options for you:

• Data grouping:

  You can follow Highstock Demos › 52,000 points with data grouping.

• Lazy Async loading:

  You can follow Highstock Demos › 1.7 million points with async loading.

The both Demos show how to fetch data by ranges and limit the chart to display only relevant range data points, which will improve time loading performances.