In distributed systems, selecting a leader is crucial to prevent multiple nodes from performing the same task. Implementing leader election using distributed locking is essential. Often, costly stateful services are used for this purpose, but with AWS S3 recently adding support for conditional writes, a simple and efficient leader election algorithm can be implemented. This algorithm involves creating lock files with unique epochs on S3 and ensuring the validity of the locks for smooth operation. Despite challenges like clock drift and potential conflicts, the leader election process can be made more robust with careful handling of lock validity and fencing tokens. This approach eliminates the need for external services and streamlines the process on AWS S3 significantly.
https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/