Facebook crawler is hitting my server hard and ignoring directives. Accessing same resources multiple times

后端 未结 8 1437
盖世英雄少女心
盖世英雄少女心 2021-02-05 05:52

The Facebook Crawler is hitting my servers multiple times every second and it seems to be ignoring both the Expires header and the og:ttl property.

In some cases, it is

8条回答
  •  说谎
    说谎 (楼主)
    2021-02-05 06:21

    Facebook documentation specifically states "Images are cached based on the URL and won't be updated unless the URL changes.". This means it doesn't matter which headers or meta tags you add to your page, the bot is supposed to cache the image anyway.

    This made me think:

    1. Does each user share a slightly different URL of your page? This will cause the share image to get re-cached each time.
    2. Is your share image accessed using a slightly different URL?
    3. Maybe the image is being linked differently somewhere?

    I'd monitor the page logs and see exactly what happens - if the page URL or the image URL is even slightly different, the caching mechanism won't work. Luckily, this doesn't seem like a headers/tags type of issue.

提交回复
热议问题