Donated Technology to Deliver Unprecedented Storage efficiency and Speed to Open Source Storage Users

San Jose, CA, February 14, 2024 – iXsystems®, the company behind TrueNAS, and Klara, a leading open source software developer, today reasserted their joint commitment to open source with the contribution of the Fast Dedup feature to the OpenZFS and TrueNAS communities. In a move sponsored and supported by engineers from iXsystems and the development efforts of Klara, the contribution of Fast Dedup significantly enhances sustained deduplication performance for ZFS and TrueNAS storage systems, marking a new milestone in storage efficiency and performance.

Deduplication performance has been limited in ZFS and OpenZFS since the inception of the file system due to its metadata handling. Fast Dedup ameliorates this with the introduction of a re-engineered metadata structure dynamically sized to operate in RAM or on dedicated flash devices for expedited processing. These advancements collectively enable up to 20 times faster performance and 5 times more usable capacity for certain workloads while greatly improving reliability for systems with deduplication enabled. The technology outperforms legacy dedup by employing algorithms and strategies that scan, identify, and eliminate duplicates, while keeping metadata in RAM and SSDs.

“The OpenZFS Community is very excited to receive this major contribution of Fast Dedup software,” said Matt Ahrens, co-creator of ZFS and the OpenZFS Project. “Fast Dedup has been a major user desire for many years and we are pleased to see that Klara and iXsystems have developed the feature in-line with our general requirements. We look forward to testing and integrating the new software.”

Key Fast Dedup Advancements Include:

• Metadata size is now dynamically sized to fit on either RAM or dedicated flash devices to avoid hitting the performance penalty wall.
• Metadata structure completely re-engineered to enable efficient updates using a log append process, greatly improving performance for large updates such as deletions.
• Dedup table now favors dedup-able data and prune blocks that show no dedup potential.
• Metadata improvements combined with properly configured storage, including dedicated metadata flash devices designed to improve sustained dedup performance by an order of magnitude or greater for larger systems.

“Klara is very proud of the speed and functional quality of the Fast Dedup development,” said Allan Jude, FreeBSD Engineering Manager at Klara “We look forward to completing the integration and testing work needed for this to be a standard OpenZFS capability, with no extra cost.”

“Fast Dedup has been a longstanding desired feature of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the performance.” said Kris Moore, SVP of iXsystems Engineering. “These attributes will significantly improve the economics of OpenZFS storage relative to Cloud storage and proprietary storage, and our team could not be more ecstatic to see this come to fruition.”

The donation of Fast Dedup technology is a testament to the shared vision and collaboration between iXsystems and Klara, underscoring their dedication to the advancement of open-source storage solutions. Expected in Q2 for TrueNAS SCALE, Fast Dedup will be available as test software, not to be used for production. This will allow validation of functionality and performance prior to a standard product release later in 2024.

For more information on iXsystems, TrueNAS, or its hardware/software products, please contact iX.

About iXsystems and TrueNAS

iX is an Open Source pioneer and the company behind TrueNAS, the world’s most deployed storage software. Relied upon by millions in all 195 countries, TrueNAS is an award-winning universal data platform used by a majority of Fortune 500TM companies. The platform harnesses the power of the legendary ZFS file system to provide scale-up or scale-out unified storage with the reliability and performance demanded by virtualization, backup, and many other data-heavy workloads. As an alternative to legacy storage systems that are proprietary, restrictive, and often overpriced, TrueNAS helps organizations modernize how they store and protect data by leveraging open storage to simplify operations and drastically reduce cost.