Celery Task Chain and Accessing **kwargs

Question

I have a situation similar to the one outlined here, except that instead of chaining tasks with multiple arguments, I want to chain tasks that return a dictionary with multi

Answer 1

This is my take at the problem, using an abstract task class:

from __future__ import absolute_import
from celery import Task
from myapp.tasks.celery import app   


class ChainedTask(Task):
    abstract = True    

    def __call__(self, *args, **kwargs):
        if len(args) == 1 and isinstance(args[0], dict):
            kwargs.update(args[0])
            args = ()
        return super(ChainedTask, self).__call__(*args, **kwargs)

@app.task(base=ChainedTask)
def task1(x, y):
    return {'x': x * 2, 'y': y * 2, 'z': x * y}    


@app.task(base=ChainedTask)
def task2(x, y, z):
    return {'x': x * 3, 'y': y * 3, 'z': z * 2}

You can now define and execute your chain as such:

from celery import chain

pipe = chain(task1.s(x=1, y=2) | task2.s())
pipe.apply_async()

Answer 2

chain and the other canvas primitives are in the family of functional utilities like map and reduce.

E.g. where map(target, items) calls target(item) for every item in the list, Python has a rarely used version of map called itertools.starmap, which instead calls target(*item).

While we could add starchain and even kwstarchain to the toolbox, these would be very specialized and probably not used as often.

Interestingly Python has made these unnecessary with the list and generator expressions, so that map is replaced with [target(item) for item in item] and starmap with [target(*item) for item in item].

So instead of implementing several alternatives for each primitive I think we should focus on finding a more flexible way of supporting this, e.g. like having celery powered generator expressions (if possible, and if not something similarly powerful)

Answer 3

Since this isn't built into celery, I wrote a decorator function to something similar myself.

# Use this wrapper with functions in chains that return a tuple. The
# next function in the chain will get called with that the contents of
# tuple as (first) positional args, rather than just as just the first
# arg. Note that both the sending and receiving function must have
# this wrapper, which goes between the @task decorator and the
# function definition. This wrapper should not otherwise interfere
# when these conditions are not met.

class UnwrapMe(object):
    def __init__(self, contents):
        self.contents = contents

    def __call__(self):
        return self.contents

def wrap_for_chain(f):
    """ Too much deep magic. """
    @functools.wraps(f)
    def _wrapper(*args, **kwargs):
        if type(args[0]) == UnwrapMe:
            args = list(args[0]()) + list(args[1:])
        result = f(*args, **kwargs)

        if type(result) == tuple and current_task.request.callbacks:
            return UnwrapMe(result)
        else:
            return result
    return _wrapper

Mine unwraps like the starchain concept, but you could easily modify it to unwrap kwargs instead.