Generative AI models require specialized clusters of expensive hardware, such as H100 GPUs and fast storage, to operate effectively. However, not all clusters are equal, with many often containing faulty components that can impact performance. At Together AI, we have developed a rigorous acceptance testing process to ensure our hardware meets the highest standards of reliability and performance. We validate clusters through a hierarchical approach, from basic functionality to complex integrations, to guarantee optimal performance. Our process includes GPU validation, NVLink and NVSwitch testing, network validation, and storage performance testing using tools like DCGM Diagnostics, gpu-burn, NCCL tests, iperf3, and fio.
https://www.together.ai/blog/a-practitioners-guide-to-testing-and-running-large-gpu-clusters-for-training-generative-ai-models