In 2010, an IT support worker, known as ‘Zimmie,’ received a call from a large US specialty retailer about its VPN hardware that had stopped working. Upon investigation, Zimmie discovered that the problem was due to a certificate validation failure caused by an oversized certificate revocation list (CRL). The CRL had grown exponentially over the weekend because the certificate authority (CA) was repeatedly revoking and reissuing certificates at an astonishing rate. Furthermore, the CA’s own certificate had expired, and it was signing replacement certificates with incorrect dates, leading to further confusion. The root cause of these issues was a faulty NTP client that allowed significant time adjustments, causing the system to believe it was in the future. Resolving the problems required physically visiting numerous sites to trust the new CA multiple times. Despite the challenges, Zimmie found satisfaction in solving the mystery and believes the story resonates with people due to its intriguing nature.
https://www.theregister.com/2024/02/09/it_incident_report_the_clock/