Running an async background task in Tornado

Question

Reading the Tornado documentation, it\'s very clear how to call an async function to return a response:

class GenAsyncHandler(RequestHandler):
    @gen.corou         


        

Answer 1

I recommend using toro. It provides a relatively simple mechanism for setting up a background queue of tasks.

The following code (put in queue.py for example), starts a simple "worker()" that simply waits until there is something in his queue. If you call queue.add(function,async,*args,**kwargs) this adds an item to the queue which will wake up worker() which then kicks off the task.

I added the async parameter so that this can support background tasks wrapped in @gen.coroutine and those without.

import toro,tornado.gen
queue = toro.Queue()
@tornado.gen.coroutine
def add(function,async,*args,**kwargs):
   item = dict(function=function,async=async,args=args,kwargs=kwargs)
   yield queue.put(item)

@tornado.gen.coroutine
def worker():
   while True:
      print("worker() sleeping until I get next item")
      item = yield queue.get()
      print("worker() waking up to process: %s" % item)
      try:
         if item['async']:
            yield item['function'](*item['args'],**item['kwargs'])
         else:
            item['function'](*item['args'],**item['kwargs'])
      except Exception as e:
         print("worker() failed to run item: %s, received exception:\n%s" % (item,e))

@tornado.gen.coroutine
def start():
   yield worker()

In your main tornado app:

import queue
queue.start()

And now you can schedule a back ground task quite simply:

def my_func(arg1,somekwarg=None):
   print("in my_func() with %s %s" % (arg1,somekwarg))

queue.add(my_func,False,somearg,somekwarg=someval)