Where do search engines start crawling?

Question

What do search engine bots use as a starting point? Is it DNS look-up or do they start with some fixed list of well-know sites? Any guesses or suggestions?

Answer 1

Your question can be interpreted in two ways:

Are you asking where search engines start their crawl from in general, or where they start to crawl a particular site?

I don't know how the big players work; but if you were to make your own search engine you'd probably seed it with popular portal sites. DMOZ.org seems to be a popular starting point. Since the big players have so much more data than we do they probably start their crawls from a variety of places.

If you're asking where a SE starts to crawl your particular site, it probably has a lot to do with which of your pages are the most popular. I imagine that if you have one super popular page that lots of other sites link to, then that would be the page that SEs starts will enter from because there are so many more entry points from other sites.

Note that I am not in SEO or anything; I just studied bot and SE traffic for a while for a project I was working on.