Why the supervisor make the celery worker changing form running to starting all the time?

问题

backgroud

The system is Centos7, which have a python2.x. 1GB memory and single core.

I install python3.x , I can code python3 into python3.

The django-celery project is based on a virtualenv python3.x,and I had make it well at nginx,uwsgi,mariadb. At least,I think so for no error happend.

I try to use supervisor to control the django-celery's worker,like below:

command=env/bin/python project/manage.py celeryd -l INFO -n worker_%(process_num)s
numprocs=4
process_name=projects_worker_%(process_num)s
stdout_logfile=logfile.log
etderr_logfile=logfile_err.log

Also had make setting about celery events,celery beat,this part is well ,no error happend. Error comes from the part of worker.

When I keep the proces big than 1,it would run at first,when I do supervisorctl status,all are running.

But when I do the same command to see status once more times,some process status change to starting.

So I try more times,I found that:the worker's status would always change from running to starting and then changeing starting to running-- no stop.

When I check the supervisor's logfile at tmp/supervisor.log,it shows like:

exit status 1; not expected
entered runnging state,process has stayed up for > than 1 seconds(startsecs)
'project_worker_0' with pid 2284

Maybe it shows why the worker change status all the time.

What's more ,when I change the proces to 1,the worker could failed.The worker's log show me:

stale pidfile exists.Removing it

But,I did not ponit the pidfile path to worker.And,I just found the events's and beat 's pidfie at the / path,no worker's pidfile.Also ,I try find / -name *.pid to find a pidfile like worker,or celeryd,but here did not exist.

question

• firstly, I want to deploy the project , so ,did here any other way to deploy the django-celery with virtulanev's celery part?

• If here anyone can tell me how this phenomenon comes,I would better to choose supervisor to deploy the celery part. Anyone can help me about it ?

PS

Any of your thoughts may be helpful to me, best wishs!

回答1:

Finally, I solve this problem yesterday night.

about the reason

• I make the project could success running at a windows 10 system, but did no check when I change the project to centos7.+. The command:env/bin/python project/manage.py celeryd could not run success. So the supervisor would start a process which will failed soon.

• Why the command could not success? I had pip installed all the package need. But it show err below:

  Running a worker with superuser privileges when the worker accepts messages serialized with pickle is a very bad idea!

  If you really want to continue then you have to set the C_FORCE_ROOT environment variable (but please think about this before you do).

  User information: uid=0 euid=0 gid=0 egid=0

  I try to search some blog about this error, and get the anser:

  export C_FORCE_ROOT='true' # at the centos enviroument

action to solve(after meeting error like this)

• add export C_FORCE_ROOT='true' to centos's enviroment file and source it.

• check command 'env/bin/python project/manage.py celeryd ',did it run successful.

• restart the supervisord. Attention please! not supervisorctl reload,it just reload the .conf file,not the environment file. Try kill the process supervisord -c xx.conf(ps aux | grep supervisord and kill -9 process_number,be careful).

some url about the blog

• the error when just run celeryd not sucess in chinese

来源：https://stackoverflow.com/questions/48005490/why-the-supervisor-make-the-celery-worker-changing-form-running-to-starting-all