The author discusses how network packet captures and BPF were used to identify a Linux kernel performance issue causing elevated listen overflow errors on Apache web servers after the fleet upgrade to Ubuntu. Initially suspected to be a network problem, it was revealed to be due to elevated system CPU, specifically inode cgroup switching, in the Linux kernel. By disabling certain cgroup accounting features using systemd, they were able to resolve the issue. This kernel issue was unique to newer Linux versions and was not present in older CentOS, highlighting the complexity of performance debugging in modern system environments. Canonical confirmed it as a kernel issue potentially introduced by a 2021 Linux kernel commit, showing the collaborative effort towards resolving it.
https://dasl.cc/2025/01/01/debugging-our-new-linux-kernel/