Join us on a fascinating journey of scaling a 10,000 node cluster at Recall.ai, processing a mind-boggling 1TB/sec of raw video data. Unexpected audio artifacts emerged after a migration, leading to a deep dive into the infrastructure. The surprising culprit? Cloud-init logging causing SystemD and EC2 issues, ultimately affecting audio quality due to CPU starvation. A simple fix solved the problem – disable cloud-init logging. The journey to uncover the root cause involved meticulous investigation and creative problem-solving, resulting in a unique and intriguing story with a happy ending, showcasing the complex interplay of various components in a large-scale system.
https://www.recall.ai/post/debugging-audio-artifacts-caused-by-a-serial-port