DoorDash transitioned from Aurora PostgreSQL to CockroachDB, uncovering a rare “one-in-a-million” bug autofiled by Sentry’s crash reporting module. Nondeterminism’s double-edged nature in Computer Science is explored, with demonic nondeterminism making distributed systems debugging difficult due to near-impossible bug reproduction. The innovative Antithesis platform, inspired by FoundationDB, simulates deterministic failures to reproduce bugs autonomously and efficiently. The root cause of the complex bug in CockroachDB is traced to an ambiguous write, requiring the implementation of a fix to ensure transaction idempotency. The Antithesis platform represents groundbreaking progress in deterministic debugging, signaling a promising future for bug detection.
https://www.cockroachlabs.com/blog/demonic-nondeterminism/