Why isn't my custom delivered image caching in the browser?

I have a custom handler that is returning an image to the browser.

The images are fetched from a database.

For some reason the images are not being cached by the browser, and I was wondering if someone might be able to spot what I am missing from the below code:

HttpContext.Current.Response.BinaryWrite(imageBytes);
HttpContext.Current.Response.Cache.SetCacheability(HttpCacheability.Public);
Context.Current.Response.Cache.SetAllowResponseInBrowserHistory(true);
if(imgRepGetCache.DateCached.HasValue)
    HttpContext.Current.Response.Cache.SetLastModified(imgRepGetCache.DateCached.Value);
HttpContext.Current.Response.Cache.SetExpires(DateTime.Now.AddDays(2));
HttpContext.Current.Response.ContentType = "image/jpeg";

Or alternatively if I'm completely missing the point somehow and there's somewhere else I need to look.

Edit: As per request for more info:

The URL is always the same
I am testing loading the same file via standard IIS pipe and my pipe in the same browser on the same PC. The one that loads through IIS normally is cached, my file isn't.

Edit 2: After inspecting the HTTP requests/responses on the normal IIS route I think it has something to do with the ETag. The ETag (which I'm new to as of just now) seems to be a sort of checksum for the document. On subsequent requests by a browser the ETag is sent and if the server finds the ETag hasn't changed then it returns a 304 - Not Modified. All good! But I'm now setting the ETag using:

HttpContext.Current.Response.Cache.SetETag(imgRepGetCache.DateCached.ToString());

But it doesn't appear in the response. Closer...

Edit 3: I fixed it in the end after taking advantage of Firebug for some HTTP inspecting fun. I posted my solution below.

OK, I fixed it.

Here is what I did for anyone else and for my own future reference:

// Check for repeated request for the same image from a browser
if (HttpContext.Current.Request.Headers.Get("If-None-Match") == imgRepGetCache.DateCached.Value.ToString())
{
    // Return 304 - Not Modified
    HttpContext.Current.Response.Status = "304 Not Modified";
}
else
{
    if (imgRepGetCache.DateCached.HasValue)
        HttpContext.Current.Response.Headers.Set("Etag", imgRepGetCache.DateCached.Value.ToString());
    // ... do my other stuff here
}

Works a charm!

If anyone spots any potential problems here, let me know so I can update this.

To pre-empt one obvious one - I can 100% rely on the date string for identifying whether an image is new or not (in my particular scenario).

You don't mention anything in your post about it, but is this a https:// address? Browsers don't cache images and pages from https sites due to security reasons.

The things you need to worry about in the generation of the response are: