I want to overlay multiple images on my video at different times. I have user function defined as
function myFunction(clip c, int coordinateX, int from, int to)
Try to convert the video and all images to YUY2 colorspace before overlaying with "ConvertToYUY2()", this might probably decrease the amount of reconversions on every call as explained here - http://avisynth.nl/index.php/Overlay#Repeated_overlays_on_RGB_base_clip That's assuming your video and images are in RGB. You can probably try to use also YV12 colorspace, maybe it will save even more memory. Note that it will cause slight quality decrease since color channels have reduced resolution in formats other than YV24 plus there are conversion errors too.
Try to play with SetMemoryMax() function. But make sure that you have enough of free memory.
Also note if you are working in 32-bit environment, max memory volume will be 2 GB.