How to efficiently serve massive sitemaps in django

Question

I have a site with about 150K pages in its sitemap. I\'m using the sitemap index generator to make the sitemaps, but really, I need a way of caching it, because building the 150

Answer 1

I had a similar issue and decided to use django to write the sitemap files to disk in the static media and have the webserver serve them. I made the call to regenerate the sitemap every couple of hours since my content wasn't changing more often than that. But it will depend on your content how often you need to write the files.

I used a django custom command with a cron job, but curl with a cron job is easier.

Here's how I use curl, and I have apache send /sitemap.xml as a static file, not through django:

curl -o /path/sitemap.xml http://example.com/generate/sitemap.xml