php spider breaks in middle (Domdocument, xpath, curl) - help needed
问题 I am a beginner programmer, designing a spider that crawls pages. Logic goes like this: get $url with curl create dom document parsing out href tags using xpath storing href attributes in $totalurls (that aren't already there) updating $url from $totalurls Problem is that after the 10th crawled page the spider says it does not find ANY links on the page, no no one on the next, and so on. But if I begin with the page that was 10th in previous example it finds all links with no problem but