Varnish ban is added but old object is returned

可紊 提交于 2021-02-08 06:32:49

问题


I'm using varnish in front of a tile server to cache mapbox tiles. To remove old tiles, I intended to use bans to effectively remove a large number of cached tiles. My problem is that varnish still uses the cached objects (at least the age in the response indicates this) and doesn't contact the backend.

I'm first requesting http://varnish/5/3/4.pbf, then adding a ban with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish or alternatively varnishadm and then ban obj.http.url ~ 5/3/4.pbf and afterwards requesting http://varnish/5/3/4.pbf again.

In the beginning my ban list is empty:

Present bans:
1610117471.434488     1 C

The ban is added succesfully with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish

<!DOCTYPE html>
<html>
  <head>
    <title>200 Ban added</title>
  </head>
  <body>
    <h1>Error 200 Ban added</h1>
    <p>Ban added</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 8</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

and shows up in the ban list

Present bans:
1610117369.028870     0 -  obj.http.url ~ 5/3/4.pbf
1610117307.220739     1 C  

After requesting http://varnish/5/3/4.pbf again, the ban list indicates that the ban was used

Present bans:
1610117471.434488     1 -  obj.http.url ~ 5/3/4.pbf

but the age of the response is not 0, because it's still the object from the first request.

After a short time the ban is removed:

Present bans:
1610117471.434488     1 C  

My vcl_recv looks like this but the error is probably somewhere else as it also doesn't work with varnishadm:

sub vcl_recv {
    unset req.http.cookie;

    # Allowing PURGE from localhost
    if (req.method == "BAN"||req.method == "PURGE") {
                if (!client.ip ~ purge) {
                        return(synth(405,"Not allowed."));
                }
                if (req.method == "BAN") {
                    ban("obj.http.url ~ " + req.http.X-Purge-Regex);

                    # Throw a synthetic page so the
                    # request won't go to the backend.
                    return(synth(200, "Ban added"));
                }
                if (req.method == "PURGE") {
                    return (purge);
                }
        }
}

I also tried to use the vcl_purge from https://stackoverflow.com/a/61507014 but this doesn't seem to help for bans (?).

I'm using the X-Purge-Regex header to not worry about having to escape special characters like in https://stackoverflow.com/a/38526921 but just a ban like obj.http.url ~ 0 doesn't work.

I'm using varnish 6.5 with vcl 4.0.

Ban request

*   << Request  >> 54        
-   Begin          req 53 rxreq
-   Timestamp      Start: 1610121483.345437 0.000000 0.000000
-   Timestamp      Req: 1610121483.345437 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       192.168.48.2 50882 http
-   ReqMethod      BAN
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: varnish-volatile
-   ReqHeader      User-Agent: curl/7.64.0
-   ReqHeader      Accept: */*
-   ReqHeader      X-Purge-Regex: 0
-   ReqHeader      X-Forwarded-For: 192.168.48.2
-   VCL_call       RECV
-   VCL_acl        MATCH purge "importer"
-   VCL_return     synth
-   VCL_call       HASH
-   VCL_return     lookup
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     Ban added
-   RespHeader     Date: Fri, 08 Jan 2021 15:58:03 GMT
-   RespHeader     Server: Varnish
-   RespHeader     X-Varnish: 54
-   VCL_call       SYNTH
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   VCL_return     deliver
-   Timestamp      Process: 1610121483.347281 0.001844 0.001844
-   RespHeader     Content-Length: 246
-   Storage        malloc Transient
-   Filters        
-   RespHeader     Accept-Ranges: bytes
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1610121483.347557 0.002120 0.000276
-   ReqAcct        98 0 98 218 246 464
-   End            

GET after adding ban

*   << Request  >> 32806     
-   Begin          req 32805 rxreq
-   Timestamp      Start: 1610121552.733872 0.000000 0.000000
-   Timestamp      Req: 1610121552.733872 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       192.168.48.1 55176 http
-   ReqMethod      GET
-   ReqURL         /public.snow_db/0/0/0.pbf
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: localhost:8090
-   ReqHeader      User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
-   ReqHeader      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
-   ReqHeader      Accept-Language: en-US,en;q=0.5
-   ReqHeader      Accept-Encoding: gzip, deflate
-   ReqHeader      DNT: 1
-   ReqHeader      Connection: keep-alive
-   ReqHeader      Upgrade-Insecure-Requests: 1
-   ReqHeader      Pragma: no-cache
-   ReqHeader      Cache-Control: no-cache
-   ReqHeader      X-Forwarded-For: 192.168.48.1
-   VCL_call       RECV
-   ReqUnset       Host: localhost:8090
-   ReqHeader      host: localhost:8090
-   VCL_return     hash
-   ReqUnset       Accept-Encoding: gzip, deflate
-   ReqHeader      Accept-Encoding: gzip
-   VCL_call       HASH
-   VCL_return     lookup
-   Hit            28 601789.331504 10.000000 0.000000
-   VCL_call       HIT
-   VCL_return     deliver
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     content-encoding: gzip
-   RespHeader     content-type: application/x-protobuf
-   RespHeader     date: Fri, 08 Jan 2021 15:09:02 GMT
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     X-Varnish: 32806 28
-   RespHeader     Age: 3010
-   RespHeader     Via: 1.1 varnish (Varnish/6.5)
-   VCL_call       DELIVER
-   VCL_return     deliver
-   Timestamp      Process: 1610121552.734070 0.000197 0.000197
-   Filters        
-   RespHeader     Accept-Ranges: bytes
-   RespHeader     Content-Length: 295
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1610121552.734217 0.000345 0.000147
-   ReqAcct        414 0 414 272 295 567
-   End  

Reproducing the issue

To reproduce the bug:

  • git clone https://github.com/Baschdl/varnish-ban-setup.git && cd varnish-ban-setup
  • docker-compose up
  • Open http://localhost:8092/5/3/1.pbf
  • docker-compose exec varnish varnishadm ban obj.http.url ~ pbf
  • Open http://localhost:8092/5/3/1.pbf again and you will get the old object

回答1:


The obj.http.url ~ 5/3/4.pbf ban that you are issuing, is matching a url response header.

Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.

Ban lurker

The ban lurker is a thread that asynchronously processes bans on the ban list and matches objects to the bans in order to remove patterns of objects from the cache.

The ban lurker doesn't operate within a request scope, but only is aware of the object scope.

In order to successfully match request information, request context can be added as a response header. And that's what you're doing via obj.http.url

So why doesn't the ban work?

The reason why your ban isn't working, is because you didn't set obj.http.url in your VCL file. As a result, the ban lurker cannot match any objects to it.

How to fix the issue

The solution is simple: set the missing headers in the backend response context, as illustrated below:

sub vcl_backend_response {
    set beresp.http.url = bereq.url;
    set beresp.http.host = bereq.http.host;
}

When the backend responds, and right before the object gets stored in cache, we can set the missing headers.

After that, the ban lurker will be able to match the ban expression to the right objects, and remove them from cache.

Don't forget the objects aren't immediately matched: they are only removed when they reach the ban_lurker_age, which is set to 1 minute by default.



来源:https://stackoverflow.com/questions/65632023/varnish-ban-is-added-but-old-object-is-returned

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!