Nvidia gives long-haul InfiniBand systems a 400G boost • The Register

SC22 Nvidia says users can now extend their 400 Gbps InfiniBand networks even further with the launch of its MetroX 3 long-haul system, which extends the range of its Quantum 2 switches to 25 miles or 40 kilometers.

Nvidia sees two main use cases for the technology: the first is high-speed workload migration between physically separate data centers, while the other involves pooling those resources to tackle larger problems.

While in the past it was possible to connect two data centers together via InfiniBand using the MetroX-2 platform acquired from Mellanox, the appliance was limited to a pair of 100Gbps uplinks. In comparison, the third iteration of Nvidia’s MetroX platform adds support for a pair of 100 Gbps Dense Wave Division Multiplex (DWDM) modules. The technology enables much higher bandwidths by multiplexing multiple 100 Gbps signals onto a single fiber.

“This allows your extended campus clusters and core data center to truly behave as one – a single data center,” said Dion Harris, director of product marketing for data centers at Nvidia, during a news conference.

The approach is not uncompromising. MetroX-3 is clearly an ecosystem game. It is designed to work with Nvidia’s InfiniBand ecosystem of Quantum 2 switches, ConnectX-7 NICs and/or BlueField data processing units (DPUs). That said, if you’re already using something like HPE’s Ethernet-based Slingshot connections, or even Nvidia’s own Spectrum switches, MetroX-3 isn’t for you.

Assuming you live in Nvidia’s walled garden or want to expand an existing InfiniBand environment in a new location, there are also performance trade-offs to consider. While DWDM enables enormous aggregate bandwidths over a single fiber, the technology is limited to relatively short runs in the neighborhood of 40 to 80 kilometers. As is usual with optics, for gain in distance you give up effective bandwidth and vice versa.

Switching to DWDM offers a cost advantage. According to Dell’Oro analyst Jimmy Yu, the more you can cram onto a single strand of fiber, the less you have to spend on fiber rentals to get a given amount of bandwidth.

And with DWDM already being widely adopted by major telcos AT&T and Lumen Technology, Nvidia says customers can now leverage existing fiber optic infrastructure rather than needing a dedicated fiber connection.

Cut through the noise

MetroX-3 is part of a broader suite of hardware and software announced by Nvidia this week at the Supercomputing event aimed at handling the growing volume of streaming data at the edge.

“By creating more high-fidelity research and instrumentation, it means you need a much more efficient way to collect, analyze and process that data,” Harris said. If “you produce 50x to 1000x more data, how much do you keep? How much do you shift back to the core? How much do you analyze.”

In addition to connecting data centers via InfiniBand, Nvidia is positioning MetroX-3 along with its Quantum series switches and BlueField DPUs as a means of extending InfiniBand networks to lab environments where the majority of data is generated. That way, the company says, customers can use their Holoscan HPC framework, running on IGX, DGX, or HGX platforms at the edge, to sift out meaningful data from the noise before feeding that refined data set back to the core data center .

Originally launched this fall alongside Nvidia’s IGX robotics and edge computing platform, Holoscan AI inference was intended for medical imaging. However, the platform has since been repurposed for use in a variety of streaming data formats, including non-image formats. Holoscan has also been overhauled to support C++ and Python APIs, which Nvidia says researchers can use to develop custom data pipelines around their workflows. ®