Facebook crawler is hitting my server hard and ignoring directives. Accessing same resources multiple times

后端 未结 8 1431
盖世英雄少女心
盖世英雄少女心 2021-02-05 05:52

The Facebook Crawler is hitting my servers multiple times every second and it seems to be ignoring both the Expires header and the og:ttl property.

In some cases, it is

8条回答
  •  死守一世寂寞
    2021-02-05 06:31

    It would appear the Facebook's crawlers aren't always that respectful. In the past we've implemented the suggestion here: excessive traffic from facebookexternalhit bot.

    It's not the best solution as it would be nice for Facebook to limit the rate of requesting but clearly they don't do that.

提交回复
热议问题