python celery: How to append a task to an old chain

问题

I keep in my database, the reference to a chain.

from tasks import t1, t2, t3
from celery import chain
res = chain(t1.s(123)|t2.s()|t3.s())()
res.get()

How can I append an other task to this particular chain ?

res.append(t2.s())

My goal is to be sure that chains are executed in the same order I specified in my code. And if a task fail in my chain, the following tasks are not executed.

For know I'm using super big tasks in a specify queue.

回答1:

All the information is contained in the message.

Messages can be in transit, maybe on the other side of the world, or they may be consumed by an intermediate processor. For this reason it's not possible to modify a message after it has been sent.

See http://docs.celeryproject.org/en/latest/userguide/tasks.html#state

My goal is to be sure that chains are executed in the same order I specified in my code. And if a task fail in my chain, the following tasks are not executed.

You can be sure of that, the order is sent as part of the message and it will not continue should any of the tasks fail.

Now, if you really want to be able to add tasks at runtime then you could store the information in a database and have the task itself check that and call the new tasks. There are some challenges when doing this though:

1) The first task in the chain will call the next task if it succeeds, then the next task will call the next task after that and so on.

2) If you add a task to this process, what happens if the first task already executed? or the second, or the third?

So as you may guess this will require some heavy synchronization to be able to work.

I guess an easier solution would be to create a task that waits for one task to complete then apply a callback:

from celery import subtask
from celery.result import from_serializable

@app.task(bind=True)
def after_task(self, result, callback, errback=None):
    result = from_serializable(result)
    if not result.ready():
        raise self.retry(countdown=1)
    if task.successful():
        subtask(callback).delay(result.get())
    else:
        if errback:
            subtask(errback)()


def add_to_chain(result, callback, errback=None):
    callback = callback.clone()     # do not modify caller
    new_result = callback.freeze()  # sets id for callback, returns AsyncResult
    new_result.parent = result
    after_task.delay(result.serializable(), callback, errback)
    return new_result

Then you can use it like this:

from tasks import t1, t2, t3

res = (t1.s(123) | t2.s() | t3.s())()
res = add_to_chain(t2.s())

NOTES:

bind=True is new in the upcoming 3.1 version, for older versions you must remove the self argument and use current_task.retry (get this from celery import current_task).

Signature.freeze is also new in 3.1, to do the same in older versions you can use:

from celery import uuid

def freeze(sig, _id=None):
    opts = sig.options
    try:
        tid = opts['task_id']
    except KeyError:
        tid = opts['task_id'] = _id or uuid()
    return sig.AsyncResult(tid)

来源：https://stackoverflow.com/questions/19400305/python-celery-how-to-append-a-task-to-an-old-chain

标签

django

RabbitMQ

celery

chain