how to avoid duplicate download urls in my python spider program?

Question

I wrote a spider program with python. It can recursively crawl web pages. I want to avoid download the same pages, so I store the urls in a list as follow:

urls          


        

Answer 1

In general, as you iterate over your URL results you could store them in a dictionary. The key of this dictionary would be the url, the value could be a boolean if you've seen the url before. In the end print the keys of this dict and it would have all unique urls.

Also, doing the lookup via a dict will give you O(1) time when checking if the URL has been seen or not.

# Store mapping of {URL: Bool}
url_map = {}

# Iterate over url results
for url in URLs:
    if not url_map.get(url, False):
        url_map[url] = True

# Values of dict will have all unique urls 
print(url_maps.keys())