Trouble Scraping Web Page With Malformed Content

若如初见. 提交于 2020-01-25 23:13:51

问题


I have written c# code which utilizes the HtmlAgilityPack library in order to scrape a page located at: World's Largest Urban Areas (Page 2). Unfortunately the page consists of malformed content.

I'm at an impasse on how to scrape this page. The current code I have (appearing below) freezes on parsing the HTML:

 HtmlNodeCollection cityRecords = _htmlDocument.DocumentNode.SelectNodes("//table[@class='boldtable']//tr[position() != 1]");
 CityNodes = (from node in cityRecords.Descendants()
              where node.Name == "td"
              select node).ToList();

The goal is to parse each and every city listed on the page with each of the data points; nothing more. Looking for recommendations on how to modify the above code or use another freely available library.

Thanks!


回答1:


Run the content through HTML Tidy before parsing it.

http://tidy.sourceforge.net/



来源:https://stackoverflow.com/questions/1908378/trouble-scraping-web-page-with-malformed-content

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!