How to run celery on windows?

Question

How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker?

Answer 1

It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker worked well.

Answer 2

After feeling like killing myself by using celery 4.4 on windows, I think I can answer this question.

For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

Then run celery worker command with default pool option.

celery worker -A <celery_file> -l info

This will run celery worker concurrently with multiple child processes.

Note: When you run celery with gevent or eventlet pool, it will work but it won't run concurrent processes on windows.

Answer 3

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose.

Use eventlet instead as below:

pip install eventlet
celery -A <module> worker -l info -P eventlet

It works for me on window 10 + celery 4.1 + python 3.

This solution solved the following exception:

[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\billiard\pool.py", line 358, in workloop
    result = (True, prepare_result(fun(*args, **kwargs)))
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\celery\app\trace.py", line 525, in _fast_trace_task
    tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack

===== update 2018-11 =====

Eventlet has an issue on subprocess.CalledProcessError:

https://github.com/celery/celery/issues/4063

https://github.com/eventlet/eventlet/issues/357

https://github.com/eventlet/eventlet/issues/413

So try gevent instead.

pip install gevent
celery -A <module> worker -l info -P gevent

This works for me on window 10 + celery 4.2 + python 3.6

Answer 4

You can still use celery 4 0+ with Windows 10+ Just use this command "celery -A projet worker - -pool=solo - l info" instead of "celery - A project worker -l info

Answer 5

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose. you can use any of this:

celery worker --app=app.app --pool=eventlet --loglevel=INFO

celery worker --app=app.app --pool=gevent --loglevel=INFO

celery worker --app=app.app --pool=solo --loglevel=INFO

or using other format:

celery -A <app> worker --loglevel=info -P eventlet

celery -A <app> worker --loglevel=info -P gevent 

celery -A <app> worker --loglevel=info -P solo

if you have get context error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo

https://www.distributedpython.com/2018/08/21/celery-4-windows/

Answer 6

yes:

celery -A your_application -l info

also note Celery have dropped support for Windows(since v4), so best to

pip install celery==3.1.25

3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn't need to downgrade flower(browser monitor for celery) though.

See also the FAQ for Windows