Celery

What Celery is

Celery is a free, open-source distributed task queue for real-time processing that also supports scheduled work. It routes task messages through a broker to worker processes, and deployments can use multiple workers and brokers for high availability and horizontal scaling. Celery Beat starts tasks at intervals for available worker nodes. Documented transports include RabbitMQ, Redis, Amazon SQS, Kafka, and Google Pub/Sub, with different capabilities: RabbitMQ and Redis are marked stable, SQS stable without monitoring or remote control, and Kafka experimental without those features. Celery documents integrations with Django, Pyramid, Pylons, Flask, web2py, Tornado, and FastAPI. Commands inspect and manage workers; monitoring options include Flower and event utilities. Celery 5.6 supports Python 3.9 through 3.13 and PyPy 3.9 or newer. It is self-hosted, with installation via PyPI or source, and is licensed under the New BSD License. The project advises restricting broker access because Celery should be treated as unsafe; supported broker backends can use SSL encryption and authentication.

Who it suits

Celery suits Python teams building self-hosted systems that need distributed task processing or scheduled jobs. Its broker and web-framework integrations may fit projects using the documented technologies.

Limits

Windows is not officially supported. Kafka is experimental without monitoring or remote control, while SQS is stable without those capabilities.

Getting started

1. Open the Celery documentation website.
2. Install Celery through PyPI or from source.
3. Choose a documented broker transport, such as RabbitMQ or Redis.
4. Configure worker processes to receive and execute task messages.
5. Use Celery Beat to start recurring tasks when scheduling is needed.

Verdict

Celery offers a self-hosted queue with scheduling, worker management, and several documented transports. Teams should account for broker-specific capability limits and the project's security guidance.