Async spawing of processes: design question - Celery or Twisted

Question

All: I\'m seeking input/guidance/and design ideas. My goal is to find a lean but reliable way to take XML payload from an HTTP POST (no problems with this part), parse it, and s

Answer 1

I Add, quite late another possibility: using Redis. Currently I using redis with twisted : I distribute work to worker. They perform work and return result asynchronously.

The "List" type is very useful : http://www.redis.io/commands/rpoplpush

So you can use the Reliable queue Pattern to send work and having a process that block/wait until he have a new work to do(a new message coming in queue.

you can use several worker on the same queue.

Redis have a low memory foot print but be careful of number of pending message , that will increase the memory that Redis use.