This section explains how to identify the NICs in your nodes and choose the correct transceivers and cables.
Step 1: Identify the NICs in Your Nodes
Most Qumulo-certified nodes are compatible with multiple NIC models. The NIC model determines transceiver compatibility.
-
Use SSH to connect to your node.
-
Run the
lspci | grep "Ethernet controller"command.Note
This command might return information about unused NICs that have interfaces with speeds of 10 Gbps (or slower).A list of NICs appears. In the following example, we ran the command on a Supermicro 1114S node, which has two ConnectX-6 NICs.
45:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01) 45:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01) 81:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6] 81:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6] c5:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6] c5:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6] -
To determine the speed and firmware compatibility information for the NICs in your node, refer to the following table.
NIC Model Speed Firmware Compatibility Information 82599ES 10 Gbps Intel 82599ES 10 Gigabit Ethernet Controller AOC-S100G-b2C 100 Gbps - Supermicro Networking Cables and Transceivers Compatibility Matrix
- Broadcom Optical Transceivers
- Supported Cables for Broadcom Ethernet Network Adapters in the Broadcom Ethernet Network Adapter User Guide
AOC-S25G-b2S 25 Gbps ConnectX-3 10 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Firmware Release Notes ConnectX-3 Pro 40 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Pro Firmware Release Notes ConnectX-4 40 Gbps Firmware Compatible Products in the NVIDIA Mellanox ConnectX-4 Adapter Cards Firmware Release Notes ConnectX-4 Lx 25 Gbps Firmware Compatible Products in the NVIDIA ConnectX-4 Lx Adapter Cards Firmware Release Notes ConnectX-5 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes ConnectX-6 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Adapter Cards Firmware Release Notes ConnectX-6 Dx 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Dx Adapter Cards Firmware Release Notes E810-CQDA2 100 Gbps Intel Ethernet Network Adapter E810-2CQDA2
Note
Intel might support, but doesn’t verify, third-party transceiver compatibility.E810-XXVDA2 25 Gbps Intel Ethernet Network Adapter E810-XXVDA2
Note
Intel might support, but doesn’t verify, third-party transceiver compatibility.P2100G 100 Gbps - Broadcom Optical Transceivers
- Supported Cables for Broadcom Ethernet Network Adapters in the Broadcom Ethernet Network Adapter User Guide
P225P 25 Gbps
Step 2: Choose Transceivers for Your Nodes
This section lists and explains the differences between the types of transceivers available for your nodes.
-
Lucent Connector (LC): The LC with two fibers is very common for 10 Gbps and 25 Gbps connections.
Note
Although there are transceivers that can use LC fiber optic cables for 40 Gbps and 100 Gbps connections, these transceivers are generally more expensive, consume more power, and are mainly intended for reusing LC cabling, or for long-distance applications. -
Lucent Connector Duplex (LC Duplex): The LC duplex with two fibers is the most common standard for 25 Gbps connections. The maximum short-range connection is 100 m and long-range connection is 10 km. There is also an extended-range standard with a maximum of 40 km.
-
Multi-Fiber Push On (MPO): The MPO connector with eight fibers is a common connector for 40 Gbps connections.
-
PAM4: Some newer switches can establish 100 Gbps connections by using double 50 Gbps PAM4 connections instead of the more common four 25 Gbps connections. For information about configuring Pulse Amplitude Modulation 4-level (PAM4), see Auto-Negotiation on Ethernet NIC Controllers in the Broadcom documentation.
-
SR4: The SR4, with four QSFP28 connections over an eight-fiber cable, is the most common and cost-efficient standard for 100 Gbps connections. The maximum range for SR4 is 100 m.
Step 3: Choose Cables for Your Transceivers
This section lists and explains the differences between the types of cables available for your transceivers.
If you use DAC or AOC cables, ensure that the manufacturers of your NIC and network switch both support your cables.
-
Optical Cables: We recommend using optical cables and optical transceivers that both the NIC and the switch support.
-
Direct Attach Cables (DACs): Although these cables are significantly cheaper than optical cables and are less prone to compatibility and thermal issues, they are limited in length (2-3 m, up to 5 m maximum).
-
Active Optical Cables (AOCs): Although these cables are cheaper than dedicated transceivers and fiber optic cables, they might cause compatibility issues, or your NIC or switch might not support them.