📓 Add doc on how it differs from celery

[jd-solanki]

Hi 👋🏻

I'm already using postgres and would love to replace celery with this one. It'll be easier for us to migrate to pgqueuer if you can share how it diferes from celery and in which scenarios it's good for us to migrate to avoid extra overhead of celery (and celery beat).

[janbjorge]

Hi 👋🏻,

Thanks for your interest in PGQueuer! I’ll do my best to provide a fair comparison between PGQueuer and Celery, though keep in mind my views might be a bit opinionated as the author of PGQueuer.

How PGQueuer Differs from Celery

1. PostgreSQL-Native:

   • PGQueuer is designed to integrate directly with PostgreSQL, using features like LISTEN/NOTIFY and FOR UPDATE SKIP LOCKED for efficient job handling.
   • Unlike Celery, which requires an external broker (e.g., Redis, RabbitMQ), PGQueuer eliminates the need for additional infrastructure, simplifying your stack.

2. Simplified Architecture:

   • If avoiding the complexity of managing extra infrastructure is a priority, PGQueuer offers a straightforward alternative.

3. Recurring Task Scheduling:

   • PGQueuer has built-in support for cron-like recurring tasks, so there’s no need for tools like celery-beat.

4. Lightweight Design:

   • PGQueuer focuses on being minimalist, making it easier to integrate into existing projects without the overhead of additional components.

Acknowledging Celery’s Strengths

Celery is a more mature project, and while I don’t have benchmarks comparing PGQueuer to Celery, I would expect Celery to be more performant and feature-rich, especially for advanced workflows that require task chaining, retries with complex policies, or integration with non-Python systems.

When PGQueuer is a Good Fit

• You're Already Using PostgreSQL: PGQueuer seamlessly integrates with your database without requiring an external broker.
• You Value Stack Simplicity: For teams that prefer fewer moving parts, PGQueuer can reduce operational complexity.
• Lightweight Recurring Jobs: If you need recurring task support without additional tools, PGQueuer provides this natively.

If your workflow prioritizes simplicity and you’re already using PostgreSQL, PGQueuer might be a great fit. That said, if your needs are more complex or involve large-scale distributed systems, Celery’s maturity and features might make it the better choice.

Feel free to share more about your use case or ask any questions—I’d love to help you explore if PGQueuer aligns with your needs.

[andrew222651]

How is PGQueuer different from Procrastinate?

[janbjorge]

How is PGQueuer different from Procrastinate?

Hi @amacfie,

Thanks for your question! While I don’t have hands-on experience with Procrastinate, I can share some high-level differences based on its documentation. Please note that, as the author of PGQueuer, my perspective might be opinionated. 😊

Key Differences

1. Design Philosophy:

   • PGQueuer is minimalist, tightly integrates with PostgreSQL, and avoids abstractions.
   • Procrastinate provides higher-level abstractions and decorators for tasks, which may suit teams preferring out-of-the-box simplicity.

2. Recurring Tasks:

   • PGQueuer includes native cron-like task scheduling.
   • Procrastinate doesn’t handle recurring tasks natively, requiring external scheduling tools.

3. Focus on Simplicity:

   • PGQueuer minimizes dependencies and is tailored for teams already using PostgreSQL.
   • Procrastinate offers more built-in features for task retries and async workflows.

If you’re seeking a PostgreSQL-native, minimal job queue with built-in recurring tasks, PGQueuer might be a great fit. For higher-level abstractions or asyncio support, Procrastinate could be a better choice.

Best,
JB

[andrew222651]

Thanks! (Although I would note that Procrastinate does have scheduled task support: https://procrastinate.readthedocs.io/en/stable/howto/advanced/cron.html). I'm curious about any performance differences -- when a new job is submitted in PGQueuer, do all available workers attempt to claim the job or are jobs scheduled to particular workers? I think Procrastinate does the former.

[janbjorge]

Thanks! (Although I would note that Procrastinate does have scheduled task support: https://procrastinate.readthedocs.io/en/stable/howto/advanced/cron.html). I'm curious about any performance differences -- when a new job is submitted in PGQueuer, do all available workers attempt to claim the job or are jobs scheduled to particular workers? I think Procrastinate does the former.

Thanks for pointing that out! Based on [Procrastinate's documentation](https://procrastinate.readthedocs.io/en/stable/discussions.html#how-does-this-all-work), the underlying approach seems very similar, but there are a few important details that likely differ.

PGQueuer:

• When a new job is submitted, all available workers attempt to claim it by querying the database using FOR UPDATE SKIP LOCKED. This ensures that only one worker locks and processes the job, avoiding contention.
• Idle workers poll the database periodically to check for new or unlocked jobs. This can be fine-tuned using worker configurations to optimize performance in high-load or idle conditions.
• Locks in PGQueuer operate at the database level, making them lightweight and efficient but without built-in support for grouped or sequential locks (as seen in Procrastinate).

Procrastinate:

• Procrastinate employs a similar mechanism where all workers listen for new tasks and compete to claim them using SELECT FOR UPDATE ([source](https://procrastinate.readthedocs.io/en/stable/discussions.html#how-does-this-all-work)).
• Procrastinate adds a layer of flexibility with locks, allowing jobs to be grouped by a shared resource (source). This ensures that tasks with the same lock run sequentially and in order, which can be particularly useful for workflows requiring strict sequential execution.

[andrew222651]

Ok thanks!