Where do search engines start crawling?

后端 未结 3 2374
死守一世寂寞
死守一世寂寞 2021-02-19 23:54

What do search engine bots use as a starting point? Is it DNS look-up or do they start with some fixed list of well-know sites? Any guesses or suggestions?

3条回答
  •  自闭症患者
    2021-02-20 00:15

    In principle they start with nothing. Only when somebody explicitly tells them to include their website they can start crawling this site and use the links on that site to search more.

    However, in practice the creator(s) of a search engine will put in some arbitrary sites they can think of. For example, their own blogs or the sites they have in their bookmarks.

    In theory one could also just pick some random adresses and see if there is a website there. I doubt anyone does this though; the above method will work just fine and does not require extra coding just to bootstrap the search engine.

提交回复
热议问题