Google App Engine - Caching generated HTML

Question

I have written a Google App Engine application that programatically generates a bunch of HTML code that is really the same output for each user who logs into my system, and I kn

Answer 1

Just serve a static version of your site

It's actually a lot easier than you think.

If you already have a file that contains all of the urls for your site (ex urls.py), half the work is already done.

Here's the structure:

+-/website
+--/static
+---/html
+--/app/urls.py
+--/app/routes.py
+-/deploy.py

/html is where the static files will be served from. urls.py contains a list of all the urls for your site. routes.py (if you moved the routes out of main.py) will need to be modified so you can see the dynamically generated version locally but serve the static version in production. deploy.py is your one-stop static site generator.

How you layout your urls module depends. I personally use it as a one-stop-shop to fetch all the metadata for a page but YMMV.

Example:

main = [
  { 'uri':'about-us', 'url':'/', 'template':'about-us.html', 'title':'About Us' }
]

With all of the urls for the site in a structured format it makes crawling your own site easy as pie.

The route configuration is a little more complicated. I won't go into detail because there are just too many different ways this could be accomplished. The important piece is the code required to detect whether you're running on a development or production server.

Here it is:

# Detect whether this the 'Development' server
DEV = os.environ['SERVER_SOFTWARE'].startswith('Dev')

I prefer to put this in main.py and expose it globally because I use it to turn on/off other things like logging but, once again, YMMV.

Last, you need the crawler/compiler:

import os
import sys
import urllib2
from app.urls import main

port = '8080'
local_folder = os.getcwd() + os.sep + 'static' + os.sep + 'html' + os.sep
print 'Outputting to: ' + local_folder

print '\nCompiling:'
for page in main:
  http = urllib2.urlopen('http://localhost:' + port + page['url'])
  file_name = page['template']
  path = local_folder + file_name
  local_file = open(path, 'w')
  local_file.write(http.read())
  local_file.close()
  print ' - ' + file_name + ' compiled successfully...'

This is really rudimentary stuff. I was actually stunned with how easy it was when I created it. This is literally the equivalent of opening your site page-by-page in the browser, saving as html, and copying that file into the /static/html folder.

The best part is, the /html folder works like any other static folder so it will automatically be cached and the cache expiration will be the same as all the rest of your static files.

Note: This handles a site where the pages are all served from the root folder level. If you need deeper nesting of folders it'll need a slight modification to handle that.