Three years ago, we introduced Quickwit, a distributed search engine aiming to be more cost-efficient, easier to manage, and capable of handling petabytes of data than Elasticsearch. Recently, engineers at Binance, the leading cryptocurrency exchange, successfully migrated multiple petabyte-scale Elasticsearch clusters to Quickwit. Achievements included indexing 1.6 PB per day, handling 100 PB of logs, and reducing compute costs by 80%. Binance’s innovative approach involved using Quickwit’s native integration with Kafka and a unified metastore for efficient search. The migration to Quickwit proved to be a successful and economically viable solution with significant benefits for large-scale log management.
https://quickwit.io/blog/quickwit-binance-story