At PeerDB, we are dedicated to developing a Postgres-first data-movement platform that prioritizes speed and simplicity. In this blog post, we will delve into our recent shift from a pull-and-push model to a more efficient streaming approach using Go channels. This change has vastly improved performance by allowing for parallel pushing while pulling, resulting in increased pipeline efficiency. By buffering and concurrently pushing data in batches, we have achieved improved efficiency, reduced latency, and a more responsive system. We harness the power of Go channels for data synchronization, resource management, concurrent processing, and error handling. Our performance improvement has been significant, with throughput of 10-12k Transactions Per Second (TPS) and minimal lag of 1-5 seconds. Moving forward, we are exploring additional optimizations, such as spilling the record stream to disk, to enhance our system’s resilience. Our goal at PeerDB is to continue providing Postgres users with a fast, cost-effective, and hardware-efficient data movement experience.
https://blog.peerdb.io/building-a-streaming-platform-in-go-for-postgres