Prioritizing Tasks with Celery Queues
By default, Celery sends all tasks to a single queue named celery. To prioritize specific tasks (like sending a password reset email immediately vs. generating a daily report), you should use Routing and multiple Queues.
1. Defining Queues
You don't strictly need to define queues upfront; Celery creates them automatically if you route a task to them. However, it is good practice to define them in your configuration.
from celery import Celery
from kombu import Queue, Exchange
app = Celery('tasks', broker='redis://localhost:6379/0')
# Optional: Explicitly define queues
app.conf.task_queues = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('high_priority', Exchange('high_priority'), routing_key='high_priority'),
Queue('low_priority', Exchange('low_priority'), routing_key='low_priority'),
)
# Set the default queue
app.conf.task_default_queue = 'default'2. Routing Tasks
You can assign tasks to queues in two ways:
A. Automatic Routing (Configuration)
This is the cleanest method. You map tasks to queues in your Celery config.
app.conf.task_routes = {
'tasks.send_email': {'queue': 'high_priority'},
'tasks.generate_report': {'queue': 'low_priority'},
# All other tasks go to 'default'
}B. Manual Routing (At Call Time)
You can specify the queue when calling the task.
from tasks import generate_report
# Send to high priority queue just this once
generate_report.apply_async(args=['2023-10-31'], queue='high_priority')3. Consuming from Queues
Now that tasks are in different queues, you need to tell your workers how to handle them.
Strategy A: Dedicated Workers (Recommended)
Run separate worker instances for different queues. This guarantees that high-priority tasks always have dedicated resources.
Terminal 1 (High Priority Worker):
celery -A tasks worker -Q high_priority --hostname=high_worker@%hTerminal 2 (Default/Low Priority Worker):
celery -A tasks worker -Q default,low_priority --hostname=default_worker@%h