Using Redis as a Broker for Celery
Redis is a popular choice for a Celery message broker because it is fast, easy to set up, and lightweight. While RabbitMQ is often the recommended broker for complex enterprise setups due to its reliability guarantees, Redis is excellent for many use cases.
1. Installation
You need to install Celery with the Redis bundle.
pip install "celery[redis]"2. Configuration
To use Redis, you simply need to configure the broker_url in your Celery app.
Basic Setup
from celery import Celery
# Format: redis://:password@hostname:port/db_number
app = Celery('tasks', broker='redis://localhost:6379/0')
@app.task
def add(x, y):
return x + yUsing Redis as Result Backend
Redis is also commonly used to store the results of tasks (the return values).
app = Celery('tasks',
broker='redis://localhost:6379/0',
backend='redis://localhost:6379/1')Note: It is good practice to use different Redis databases (e.g., /0 and /1) for the broker and the backend to avoid key collisions, although Celery keys are usually namespaced.
3. Running Redis (via Docker)
The easiest way to get Redis running locally is using Docker.
docker run -d -p 6379:6379 --name my-redis redis4. Important Considerations
Visibility Timeout
One specific behavior of Redis as a broker is the Visibility Timeout.
If a worker picks up a task but crashes (or takes too long) before acknowledging it, Redis needs to know when to give that task to another worker.
- Default: 1 hour (3600 seconds).
- Issue: If a task is lost, it won't be retried for an hour.
- Configuration: Set
broker_transport_options.
app.conf.broker_transport_options = {
'visibility_timeout': 3600, # 1 hour
}If you have very long-running tasks, you must increase this value, otherwise, the task might be redelivered to another worker while the first one is still processing it, causing duplication.