Using Postgres-as-queue has been a successful strategy for my team. It eliminates the need for complex infrastructure/configuration, slow multi-container testing, and expertise in anything beyond Python and Postgres. It simplifies the architecture by reducing the number of moving parts. Unlike ad-hoc SNS/PubSub/Kafka architectures, we don’t face CMD-clickability issues. The main objection to using Postgres-as-queue is the concern about performance and increased database load. However, profiling can help determine if this approach is suitable for specific use cases. The article provides a hypothetical example comparing the database load of a task before and after implementing Postgres-as-queue. The figures are fictional but demonstrate the overhead. The article also discusses implementation details and bonus features, such as retrying failed tasks, using timestamps for timeouts, and performing cruft cleanup. The appendix includes a helpful shutdown handler for polling loops.
https://leontrolski.github.io/postgres-as-queue.html