block search engine crawling directory

后端 未结 1 1024
既然无缘
既然无缘 2021-01-26 11:34

My site have URL (longest) structure like this

http://www.example.com/xyz-pqr/abcd-efgh/123.html

So there is maximum of 3 directory level, but because of CMS and

1条回答
  •  被撕碎了的回忆
    2021-01-26 11:48

    I'm not certain, but I think the following should work:

    User-agent: *
    Disallow: /*/*/*/
    

    So, given these two URLs:

    http://www.example.com/xyz-pqr/abcd-efgh/123.html
    http://www.example.com/xyz-pqr/abcd-efgh/foo-bar/123.html
    

    The first would be accepted because it has only two directory segments (/xyz-pqr-abcd-efgh).

    The second would be blocked because it has three directory segments.

    And anything longer would be blocked, as well.

    0 讨论(0)
提交回复
热议问题