In this article, the author explores the concept of running multiple OpenTelemetry Collectors within an architecture and highlights the benefits of using Apache Kafka in conjunction with the collectors. The OpenTelemetry Collector serves as a crucial component in an observability deployment, gathering, compressing, managing, and filtering data before sending it to the observability backend. However, in certain cases, a single collector may not be enough. A multi-collector architecture, where one collector collects data and batches it before sending it to another central collector, offers advantages such as scalability, reduced network traffic, and the ability to perform data filtering and sampling. The author also discusses the use of a Kafka queue for reliable data ingestion and storage during database outages or traffic bursts. The combination of OpenTelemetry Collector and Apache Kafka can improve data reliability, reduce latency, and enhance scalability in observability systems.
https://signoz.io/blog/maximizing-scalability-apache-kafka-and-opentelemetry/