CSS File Not Updating on Deploy (Google AppEngine)

Question

I pushed a new version of my website, but now the CSS and static images are not deploying properly.

Here is the messed up page: http://www.gaiagps.com

Appeng

Answer 1

I've seen this before on App Engine, even when using cache-busting query parameters like /stylesheets/default.css?{{ App.Version }}.

Here's my (unconfirmed) theory:

1. You push a new version by deploying or changing a new version to default.
2. While this update is being propagated to all GAE instances running your app...
3. ...someone hits your site.
4. The request for static resource default.css{{ App.Version }} is sent to Google's CDN, which doesn't yet have it.
5. Google's CDN asks GAE for the resource before propagation from step #2 is done for all instances.
6. If you're unlucky, GAE serves up the resource from an instance running the old version...
7. ...which now gets cached in Google's CDN as the authoritative "new" version.

When this (if this is what happens) happens, I can confirm that no amount of cache-busting browser work will help. The Google CDN servers are holding the wrong version.

To fix: The only way I've found to fix this is to deploy another version. You don't run the risk of this happening again (if you haven't made any CSS changes since the race condition), because even if the race condition occurs, presumably your first update is done by the time you deploy your second one, so all instances will be serving the correct version no matter what.