Meta AI Research SuperCluster

Meta's AI Research SuperCluster (RSC), completed in early 2022, was one of the fastest AI supercomputers in the world at the time of completion. It contains 16,000 NVIDIA A100 GPUs, connected via NVIDIA's InfiniBand fabric and a custom storage system capable of 16 TB/s throughput. Meta trained the Llama model family on RSC. The cluster consumed approximately 20MW of power. Meta announced plans to expand to 35,000 H100 GPUs by end of 2023 and has since announced investments in hundreds of thousands of H100s for the Llama 3 and beyond training runs. AICI tracks the RSC as a reference point for the scale of compute required to train frontier open-weight models — and therefore for the concentration of AI capability among the handful of organisations that can build such infrastructure.

GPUs: 16,000× NVIDIA A100 | Interconnect: NVIDIA InfiniBand HDR | Storage throughput: 16 TB/s | Storage capacity: 1 exabyte (planned) | Power: ~20MW