Nvidia H100-based Henri system tests AMD’s Green500 lead • The Register

SC22 There’s a new energy-efficiency king at the top of this fall’s Green500 ranking of the world’s greenest supercomputers, and it’s a tiny 31-kilowatt cluster powered by Nvidia’s H100 GPUs.

Developed by Lenovo for the Flatiron Institute in New York, the two-petaflop Henri system is the first and only system to use Nvidia’s Hopper GPU architecture to make this fall’s supercomputing leaderboards.

The system itself is more of an HPC cluster than a supercomputer, much like Frontier or LUMI. Based on Lenovo’s ThinkSystem SR670 V2 server platform, each node pairs two 32-core Ice Lake Xeon scalable processors with four of Nvidia’s 80GB H100 GPUs. With a total of 5920 cores between the CPU and GPUs, Henri is the second smallest system on the list.

But unlike the Top500, which prioritizes raw performance, the Green500 weighs that performance against a system’s power consumption and ranks systems by how many gigaflops they can squeeze from each watt.

At just over 65 gigaflops per watt, Henri managed to zoom past Oak Ridge National Laboratory’s TDS testbed, the former efficiency champion. And this despite the fact that Henri only achieves 37.6 percent of his theoretical maximum performance of 5.4 petaflops.

This suggests there’s still a fair amount of grunt on the table, and if Flatiron or Lenovo can handle better-than-linear performance scaling, the system’s efficiency should improve over time, too.

Notes on Hopper’s efficiency

Comparing Henri to the second most efficient system, also using Intel CPUs and Nvidia GPUs – the Atos THX.AB cluster – the H100-based system is about 59 percent more efficient.

While it’s hard to say how much of this can be attributed to Nvidia’s new hopper architecture in the H100, it certainly looks promising.

As we’ve seen with systems like Australia’s Setonix, GPU acceleration has an outsized impact on efficiency compared to CPU computation. Setonix’s CPU-only configuration ranked #338 on this fall’s Green500, while its GPU-accelerated configuration finished fourth.

That being said, a larger system using the same hardware as Henri is unlikely to scale linearly. As systems get larger, it’s fairly common for performance costs to affect the overall efficiency of the system. For example, the full-size Frontier supercomputer at ORNL is 74 times larger than the Frontier TDS system, but about 16 percent less efficient.

Another unknown is the performance of Nvidia’s H100s when paired with faster, PCIe 5.0-equipped CPUs like Intel’s Sapphire Rapids.

AMD dominates Green500 again

As usual, Green500 saw some line-up changes this fall. But as we saw in this spring’s ranking, HPE’s AMD-based systems continue to power the majority of the top 10 most efficient systems.

Frontier TDS, Adastra, Setonix – GPU, Dardel – GPU, Frontier, and LUMI – now occupying places two through seven on the Green500 – are all powered by HPE’s Cray EX235a platform, AMD’s 64-core Epyc-3 -Milan CPUs combined with the chipmaker’s Instinct MI250x GPUs.

France’s No. 10-ranked Champollion system, based on HPE’s Apollo server platform, also uses AMD Epyc processors but opts for Nvidia’s A100 GPUs over Instinct.

The only non-AMD systems to make the top 10 were Atos’ THX.AB in eighth place and MN-3 in ninth place. Both systems are equipped with scalable Intel Xeon processors.

A full breakdown of the Green500 can be found here.

change on the horizon

AMD’s dominance at the top of the Green500 may not last much longer. Next-gen CPUs and GPUs from Intel, AMD, and Nvidia will find their way into systems over the next few months.

This week we learned that the Adastra system would be one of the first to use AMD’s Epyc 4 Genoa CPUs. The chips promise a 14 percent IPC uplift over 50 percent more cores, but it’s hard to say if that will be enough to maintain AMD’s lead.

Intel’s HBM-stacked Xeon processors and Ponte Vecchio GPUs — now called Xeon Max and Data Center GPU Max — are already on their way to Argonne National Labs to be integrated into the Aurora supercomputer. Los Alamos National Lab’s (LANL) Crossroads machine will also use Intel’s Xeon Max processors.

Likewise, the first supercomputers powered by Nvidia’s Grace and Grace Hopper superchips, including LANL’s Venado system, are expected to hit the market sometime next year.

Depending on how those chips perform in HPC applications, the Green500 could look very different next spring. ®