Linux - Forgetting Exec
Today, I came across a bug where a Celery worker wasn’t gracefully shutting down, and it was causing some odd “Connection Refused” errors from requests within the task being run by the worker. It was also shutting down before it could send errors to Rollbar/Sentry for the team to know they need to address it.
This was happening because the entrypoint had a script that effectively did this:
echo "starting celery worker"
celery -A tasks worker
The problem with that entrypoint: it’s not using exec to run the child process. Without exec, it will not forward signals like SIGTERM to the process the entrypoint is waiting on. The Celery worker child process won’t know it needs to shut down gracefully if the SIGTERM is not forwarded to it.
The fixed example would look like this:
echo "starting celery worker"
exec celery -A tasks worker