JuiceFS, a file system written in Go, can manage billions of files in a single namespace. Its metadata engine optimizes memory usage with techniques like memory pools and directory compression, achieving remarkable performance. The Enterprise Edition uses an in-house metadata engine for enhanced performance. Go was chosen for development due to its efficiency, performance, and portability. JuiceFS focuses on all-in-memory management and lock-free services for quick metadata processing. The system also implements multi-partition horizontal scaling. Optimizations like memory pools and manual management reduce metadata memory usage to 100 bytes per file. JuiceFS surpasses other file systems in memory efficiency, making it ideal for extreme scenarios.
https://juicefs.com/en/blog/engineering/reduce-metadata-memory-usage