The author discusses implementing a durable, distributed log using S3, highlighting the benefits of Zero Disk Architecture. Emphasizing the importance of logs in data and event streaming systems, they showcase how logs can be used to build databases, message queues, and event streaming systems. The post includes code snippets for writing and reading logs on S3, ensuring unique offsets and data integrity through checksums. The author addresses failover and crash recovery scenarios, providing solutions. They mention the open-source project s3-log, inviting contributions and discussing potential optimizations. They advocate for using S3 due to its scalability, durability, and cost-effectiveness compared to systems like Kafka.
https://avi.im/blag/2024/s3-log/