Instead of using a table plugin by someone else, I choose to do it the standard HTML way of making Tables and rows etc etc. I would fetch data from a database to fill the rows.
I haven't heard of a plugin that can do something like this, but I mostly avoid fancy JS "libraries". That being said conceptually, you can use an AJAX request to select for example images where m>id>n, then in your javascript when you want to load more images, do another ajax request, but increment m and n by 100 for example.
Lots of open source gallery frameworks exist that operate on this principle, but usually, they generate the page using php, instead of dynamically altering the inner html using AJAX.
Here is how to get database data out of AJAX and into JS. http://www.w3schools.com/ajax/ajax_database.asp
Having thousands of rows of data in the DOM is an awful idea for one simple reason: the DOM is slow.
Instead, only render the rows that need to be visible. When a row goes out of view, remove it from the DOM.
My favorite library for displaying a grid of data is SlickGrid. By only rendering DOM elements for the data in the viewport, it can display effectively unlimited amounts data with astounding performance. (That example displays a half-million rows!)
There are more examples that show off SlickGrid's features, but you'll be especially interested in the AJAX loading example, where small chunks of rows are streamed from the server as you scroll.