How to structure celery tasks

后端 未结 2 1318
庸人自扰
庸人自扰 2021-02-05 02:57

I have 2 types of task: async tasks and schedule tasks. So, here is my dir structure:

proj
  |
  -- tasks
      |
      -- __init__.py
      |
      -- celeryapp         


        
2条回答
  •  执念已碎
    2021-02-05 04:00

    Celery tasks can be async, sync or scheduled depends on its invocation

    task.delay(arg1,arg2)       #will be async
    task.delay(arg1,arg2).get() #will be sync
    task.delay(arg1,arg2).get() #will be sync
    task.apply_async(args = [arg1,arg2], {'countdown' : some_seconds}) #async with delay
    

    There's a lot of invocations depending on your needs
    However, you must start celery with -B flag to enable celery scheduler

    $ celery worker --app=tasks -B -Q my_queue,default_queue
    

    So the way you take to organize your tasks is something personal and it deppends on your project complexity, but I think that organize them by its type of synchronism wouldn't be the best option.

    I've googled this topic and I haven't found any guide or advise, but I've read some cases that organize their task by their functionality.
    I've followed this advise, because this isn't a pattern, in my projects. Here one example of how I did

    your_app
        |
        -- reports
            |
            -- __init__.py
            -- foo_report.py
            -- bar_report.py
            -- tasks
                |
                -- __init__.py
                -- report_task.py
        -- maintenance
            |
            -- __init__.py
            -- tasks
                |
                -- __init__.py
                -- delete_old_stuff_task.py
        -- twitter
            |
            -- __init__.py
            -- tasks
                |
                -- __init__.py
                -- batch_timeline.py                
    

提交回复
热议问题