Diced Celery

Some random bits of code when working with Celery

Running Celery as a daemon with Supervisor

pip install supervisor
cd /var/www/app
echo_supervisord_conf > supervisord.conf

In the supervisord.conf, add the following lines after the supervisord section

[program:celeryd]
command=/home/thomas/virtualenvs/yourvenv/bin/celery worker --app=myapp -l info 
stdout_logfile=/path/to/your/logs/celeryd.log
stderr_logfile=/path/to/your/logs/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600

To run it:

supervisord

Commands for the interactive shell:

supervisorctl tail celeryd
$ supervisorctl restart celeryd

(Copied from https://thomassileo.name/blog/2012/08/20/how-to-keep-celery-running-with-supervisor/)

List Queues

sudo rabbitmqctl list_queues
or
sudo rabbitmqctl list_queues name messages consumers

While running supervisor and you need to kill it

sudo unlink /tmp/supervisor.sock
or
sudo unlink /run/supervisor.sock
ps auxww | grep 'supervisord' | awk '{print $2}' | xargs sudo kill -9
#Show log
cat /tmp/supervisord.log

Kill you some celery workers:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

To get celery worker name:

>>> import celery; celery.current_app.control.inspect().ping()

When Linux OOM eats some processes and want to check what got killed

egrep -i 'killed process' /var/log/messages
or
dmesg | egrep -i 'killed process'

Launch Celery from the command line

celery worker -A app.celery -P gevent -c 1000 -l info -n worker1.%h

Get Async Results

from celery.result import AsyncResult
res = task.AsyncResult("task-id")
res.ready() or res.get()

Rabbit MQ:

Add user:
sudo add_user username password
set_permissions username ".*" ".*" ".*"
Delete user:
sudo delete_user username
Restart server and clear
sudo service rabbitmq-server restart
or
sudo rabbitmqctl stop_app
sudo rabbitmqctl reset
sudo rabbitmqctl start_app

If you get this error, when you try to run celery through supervisor

Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

When you run celery through supervisor, by default its going to apply the user for celery who is executing the supervisor command.

So if your username is ubuntu and you launch the supervisor daemon in your terminal, the celery process will be launched with ubuntu as the user

But sometimes this could be undesirable (for example when starting up AWS instances with user data scripts where the user would be root and hence the above shown force_root error. In this case, you could add the

user=ubuntu right after the celery program block in supervisor

[program:celeryd]
user=ubuntu
command=celery worker -A app.helpers.async_tasks.celery --loglevel=info -P gevent -c 1000