Django: how to serve user-submitted images & thumbnails from separate, multiple servers?

Question

For my Django site, I\'d like to:

1. Accept images submitted by users
2. Generate thumbnails from those images
3. Put both the original images and the thu

Answer 1

This sounds like a job for distributed task queues. My personal favorite is Beanstalkd b/c it is so lightweight and easy to use.

Server: https://github.com/kr/beanstalkd
Client Library: https://github.com/PeterScott/beanstalkc
Django Helper: https://github.com/jonasvp/django-beanstalkd

The way it would work is like this:
1. File gets uploaded and stored locally
2. Job gets created by upload view and put into a beanstalk tube
3. Beanstalk workers are watching the tubes for work, they grab the new job
4. The worker could create the thumbnails and distribute it to as many servers as you need

The beauty is that you can have as many workers feeding off the tube as is required. So if you ever get behind, simply fire up more workers. The workers can be on the same machine or on many separate machines, and it will all work fine.