debugging long running PHP script

Question

I have php script running as a cron job, extensively using third party code. Script itself has a few thousands LOC. Basically it\'s the data import / treatment script. (JSON to

Answer 1

Although it is not stipulated, and if my guess is correct you seem to be dealing with records one at a time, but in one big cron.

i.e. Grab a record#1, munge it somehow, add value to it, reformat it then save it, then move to record#2

I would consider breaking the big cron down. ie

Cron#1: grab all the records, and cache all the salient data locally (to that server). Set a flag if this stage is achieved.

Cron #2: Now you have the data you need, munge and add value, cache that output. Set a flag if this stage is achieved.

Cron #3: Reformat that data and store it. Delete all the files.

This kind of "divide and conquer" will ease your debugging woes, and lead to a better understanding of what is actually going on, and as a bonus give you the opportunity to rerun say, cron 2.

I've had to do this many times, and for me logging is the key to identifying weaknesses in your code, identify poor assumptions about data quality, and can hint at where latency is causing a problem.