OpenZFS 2.3.0, set to be released soon, introduces the “Fast Dedup” feature, a significant improvement from traditional dedup. The author, part of the Klara team, highlights the challenges of dedup, such as memory usage and performance issues, faced by users. The new version aims to address these problems by optimizing the dedup table and live entry list, making deduplication more efficient. The article provides in-depth technical explanations of how dedup works and why traditional dedup faced issues. “Fast Dedup” strives to reduce storage requirements and improve performance, offering a better solution for managing deduplication. It is noted that deduplication may still not be suitable for all workloads.
https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/