How to return data from a CherryPy BackgroundTask running as fast as possible

Question

I\'m building a web service for iterative batch processing of data using CherryPy. The ideal workflow is as follows:

1. Users POST data to the service for process

Answer 1

Don't run a background task using the BackgroundTask solution, because it will run in a thread and, due to the GIL, cherrypy won't be able to answer new requests. Use a queue solution that runs your background tasks in a different process, like Celery or RQ.

I'm going to develop in detail an example using RQ. RQ uses Redis as a message broker, so first of all you need to install and start Redis.

Then create a module (mytask in my example) with the long time running background methods:

import time
def long_running_task(value):
    time.sleep(15)
    return len(value)

Start one (or more than one if you want to run tasks in parallel) RQ workers, it's important that the python that is running your workers has access to your mytask module (export the PYTHONPATH before running the worker if your module it's not already in the path):

# rq worker

Above you have a very simple cherrypy webapp that shows how to use the RQ queue:

import cherrypy
from redis import Redis
from rq import Queue    
from mytask import long_running_task


class BackgroundTasksWeb(object):

    def __init__(self):
        self.queue = Queue(connection=Redis())
        self.jobs = []

    @cherrypy.expose
    def index(self):
        html =  ['', '']
        html += ['
', '', '', "
"]
        html += ['