HtmlAgilityPack and large HTML Documents
问题 I have built a little crawler and now when trying it out i found that when crawling certain sites my crawler uses 98-99% CPU. I used dotTrace to see what the problem could be and it pointed me towards my httpwebrequest method - i optimised it a bit with the help of some previous questions here on stackoverflow.. but the problem was still there. I then went to see what URLs that were causing the CPU load and found that it was actually sites that are extremely large in size - go figure :) So,