How to plot large data vectors accurately at all zoom levels in real time?

Question

I have large data sets (10 Hz data, so 864k points per 24 Hours) which I need to plot in real time. The idea is the user can zoom and pan into highly detailed scatter plots.

The data is not very continuous and there are spikes. Since the data set is so large, I can't plot every point each time the plot refreshes.

But I also can't just plot every nth point or else I will miss major features like large but short spikes.

Matlab does it right. You can give it a 864k vector full of zeros and just set any one point to 1 and it will plot correctly in real-time with zooms and pans.

How does Matlab do it?

My target system is Java, so I would be generating views of this plot in Swing/Java2D.

Answer 1

You should try the file from MATLAB Central:

https://mathworks.com/matlabcentral/fileexchange/15850-dsplot-downsampled-plot

From the author:

This version of "plot" will allow you to visualize data that has very large number of elements. Plotting large data set makes your graphics sluggish, but most times you don't need all of the information displayed in the plot. Your screen only has so many pixels, and your eyes won't be able to detect any information not captured on the screen.

This function will downsample the data and plot only a subset of the data, thus improving the memory requirement. When the plot is zoomed in, more information gets displayed. Some work is done to make sure that outliers are captured.

Syntax:

dsplot(x, y)  
dsplot(y)  
dsplot(x, y, numpoints)  

Example:

x =linspace(0, 2*pi, 1000000);  
y1=sin(x)+.02*cos(200*x)+0.001*sin(2000*x)+0.0001*cos(20000*x);  
dsplot(x,y1);

Answer 2

I don't know how Matlab does it, but I'd start with Quadtrees.

Dump all your data points into the quadtree, then to render at a given zoom level, you walk down the quadtree (starting with the areas that overlap what you're viewing) until you reach areas which are comparable to the size of a pixel. Stick a pixel in the middle of that area.

added: Doing your drawing with OpenGL/JOGL will also help you get faster drawing. Especially if you can predict panning, and build up the points to show in a display list or something, so that you don't have to do any CPU work for the new frames.

Answer 3

10Hz data means that you only have to plot 10 frames per second. It should be easy, since many games achieve >100 fps with much more complex graphics.

If you plot 10 pixels per second for each possible data point you can display a minute worth of data using a 600 pixel wide widget. If you save the index of the 600th to last sample it should be easy to draw only the latest data.

If you don't have a new data-point every 10th of a second you have to come up with a way to insert an interpolated data-point. Three choices come to mind:

1. Repeat the last data-point.
2. Insert an "empty" data-point. This will cause gaps in the graph.
3. Don't update the graph until the next data-point arrives. Then insert all the pixels you didn't draw at once, with linear interpolation between the data-points.

To make the animation smooth use double-buffering. If your target language supports a canvas widget it probably supports double-buffering.

When zooming you have the same three choices as above, as the zoomed data-points are not continuous even if the original data-points were.

This might help for implementing it in Java.