django static files versioning

Question

I\'m working on some universal solution for problem with static files and updates in it.

Example: let\'s say there was site with /static/styles.css file - a

Answer 1

Is reinventing the wheel and creating own implementation that bad? Furthermore I would like low level code (nginx for example) to serve my staticfiles in production instead of python application, even with backend. And one more thing: I'd like links stay the same after recalculation, so browser fetches only new files. So here's mine point of view:

template.html:

{% load md5url %}
<script src="{% md5url "example.js" %}"/>

out html:

static/example.js?v=5e52bfd3

settings.py:

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')

appname/templatetags/md5url.py:

import hashlib
import threading
from os import path
from django import template
from django.conf import settings

register = template.Library()


class UrlCache(object):
    _md5_sum = {}
    _lock = threading.Lock()

    @classmethod
    def get_md5(cls, file):
        try:
            return cls._md5_sum[file]
        except KeyError:
            with cls._lock:
                try:
                    md5 = cls.calc_md5(path.join(settings.STATIC_ROOT, file))[:8]
                    value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
                except IsADirectoryError:
                    value = settings.STATIC_URL + file
                cls._md5_sum[file] = value
                return value

    @classmethod
    def calc_md5(cls, file_path):
        with open(file_path, 'rb') as fh:
            m = hashlib.md5()
            while True:
                data = fh.read(8192)
                if not data:
                    break
                m.update(data)
            return m.hexdigest()


@register.simple_tag
def md5url(model_object):
    return UrlCache.get_md5(model_object)

Note, to apply changes an uwsgi application (to be specific a process) should be restarted.

Answer 2

Django 1.7 added ManifestStaticFilesStorage, a better alternative to CachedStaticFilesStorage that doesn't use the cache system and solves the problem of the hash being computed at runtime.

Here is an excerpt from the documentation:

CachedStaticFilesStorage isn’t recommended – in almost all cases ManifestStaticFilesStorage is a better choice. There are several performance penalties when using CachedStaticFilesStorage since a cache miss requires hashing files at runtime. Remote file storage require several round-trips to hash a file on a cache miss, as several file accesses are required to ensure that the file hash is correct in the case of nested file paths.

To use it, simply add the following line to settings.py:

STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

And then, run python manage.py collectstatic; it will append the MD5 to the name of each static file.

Answer 3

The major advantage of this solution: you dont have to modify anything in the templates.

This will add the build version into the STATIC_URL, and then the webserver will remove it with a Rewrite rule.

settings.py

# build version, it's increased with each build
VERSION_STAMP = __versionstr__.replace(".", "")
# rewrite static url to contain the number
STATIC_URL = '%sversion%s/' % (STATIC_URL, VERSION_STAMP)

So the final url would be for example this:

/static/version010/style.css

And then Nginx has a rule to rewrite it back to /static/style.css

location /static {
    alias /var/www/website/static/;
    rewrite ^(.*)/version([\.0-9]+)/(.*)$ $1/$3;
}

Answer 4

Simple templatetag vstatic that creates versioned static files urls that extends Django's behaviour:

from django.conf import settings
from django.contrib.staticfiles.templatetags.staticfiles import static

@register.simple_tag
def vstatic(path):
    url = static(path)
    static_version = getattr(settings, 'STATIC_VERSION', '')
    if static_version:
         url += '?v=' + static_version
    return url

If you want to automatically set STATIC_VERSION to the current git commit hash, you can use the following snippet (Python3 code adjust if necessary):

import subprocess


def get_current_commit_hash():
    try:
        return subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip().decode('utf-8')
    except:
        return ''

At settings.py call get_current_commit_hash(), so this will be calculated only once:

STATIC_VERSION = get_current_commit_hash()

Answer 5

I would suggest using something like django-compressor. In addition to automatically handling this type of stuff for you, it will also automatically combine and minify your files for fast page load.

Even if you don't end up using it in entirety, you can inspect their code for guidance in setting up something similar. It's been better vetted than anything you'll ever get from a simple StackOverflow answer.

Answer 6

There is an update for @deathangel908 code. Now it works well with S3 storage also (and with any other storage I think). The difference is using of static file storage for getting file content. Original doesn't work on S3.

appname/templatetags/md5url.py:

import hashlib
import threading
from django import template
from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage

register = template.Library()


class UrlCache(object):
    _md5_sum = {}
    _lock = threading.Lock()

    @classmethod
    def get_md5(cls, file):
        try:
            return cls._md5_sum[file]
        except KeyError:
            with cls._lock:
                try:
                    md5 = cls.calc_md5(file)[:8]
                    value = '%s%s?v=%s' % (settings.STATIC_URL, file, md5)
                except OSError:
                    value = settings.STATIC_URL + file
                cls._md5_sum[file] = value
                return value

    @classmethod
    def calc_md5(cls, file_path):
        with staticfiles_storage.open(file_path, 'rb') as fh:
            m = hashlib.md5()
            while True:
                data = fh.read(8192)
                if not data:
                    break
                m.update(data)
            return m.hexdigest()


@register.simple_tag
def md5url(model_object):
    return UrlCache.get_md5(model_object)