You shouldn't let your crawler run under apache, it's better to run it stand-alone via cli as part of a Gearman setup.
That way it won't hog your web server and it can run as long as you want. You can find many bindings for Gearman that you can use, including PHP of course.