In this blog post, the author describes their journey investigating a months-long issue with their network. They initially suspected the problem was with their ingress controller, but after further investigation, they discovered that the issue was actually caused by DNS. They experienced spikes in HTTP errors and noticed a correlation between errors in a fluent-bit agent and degraded service performance. After making several enhancements to their DNS configuration, such as increasing the concurrent queries limit and discarding garbage domains, they were able to improve performance and resolve the issue. The author emphasizes the importance of having solid SLIs and organizing the investigation process.
https://medium.com/adevinta-tech-blog/its-not-always-dns-unless-it-is-16858df17d3f