The interface to the chart object in Excel only allows access to the image representing the chart through two methods:
One option that may work would be to use a memory mapped file. Of course disk can be involved there, so you should profile the standard export to disk vs. using a memory mapped file (assuming the slowness of disk i/o is your only concern with that approach).
They don't have out of the box support in .Net yet, but there likely plenty of implementations out there.