This section explains how to identify the NICs in your nodes and choose the correct transceivers and cables.

Step 1: Identify the NICs in Your Nodes

Most Qumulo-certified nodes are compatible with multiple NIC models. The NIC model determines transceiver compatibility.

  1. Use SSH to connect to your node.

  2. Run the lspci | grep "Ethernet controller" command.

    A list of NICs appears. In the following example, we ran the command on a Supermicro 1114S node, which has two ConnectX-6 NICs.

    45:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
    45:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
    81:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    81:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    c5:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    c5:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
    
  3. To determine the speed and firmware compatibility information for the NICs in your node, refer to the following table.

    NIC Model Speed Firmware Compatibility Information
    82599ES 10 Gbps Intel 82599ES 10 Gigabit Ethernet Controller
    AOC-S100G-b2C 100 Gbps
    AOC-S25G-b2S 25 Gbps
    ConnectX-3 10 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Firmware Release Notes
    ConnectX-3 Pro 40 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Pro Firmware Release Notes
    ConnectX-4 40 Gbps Firmware Compatible Products in the NVIDIA Mellanox ConnectX-4 Adapter Cards Firmware Release Notes
    ConnectX-4 Lx 25 Gbps Firmware Compatible Products in the NVIDIA ConnectX-4 Lx Adapter Cards Firmware Release Notes
    ConnectX-5 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes
    ConnectX-6 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Adapter Cards Firmware Release Notes
    ConnectX-6 Dx 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Dx Adapter Cards Firmware Release Notes
    E810-CQDA2 100 Gbps

    Intel Ethernet Network Adapter E810-2CQDA2

    P2100G 100 Gbps
    P225P 25 Gbps

Step 2: Choose Transceivers for Your Nodes

This section lists and explains the differences between the types of transceivers available for your nodes.

  • Lucent Connector (LC): The LC with two fibers is very common for 10 Gbps and 25 Gbps connections.

  • Lucent Connector Duplex (LC Duplex): The LC duplex with two fibers is the most common standard for 25 Gbps connections. The maximum short-range connection is 100 m and long-range connection is 10 km. There is also an extended-range standard with a maximum of 40 km.

  • Multi-Fiber Push On (MPO): The MPO connector with eight fibers is a common connector for 40 Gbps connections.

  • PAM4: Some newer switches can establish 100 Gbps connections by using double 50 Gbps PAM4 connections instead of the more common four 25 Gbps connections. For information about configuring Pulse Amplitude Modulation 4-level (PAM4), see Auto-Negotiation on Ethernet NIC Controllers in the Broadcom documentation.

  • SR4: The SR4, with four QSFP28 connections over an eight-fiber cable, is the most common and cost-efficient standard for 100 Gbps connections. The maximum range for SR4 is 100 m.

Step 3: Choose Cables for Your Transceivers

This section lists and explains the differences between the types of cables available for your transceivers.

  • Optical Cables: We recommend using optical cables and optical transceivers that both the NIC and the switch support.

  • Direct Attach Cables (DACs): Although these cables are significantly cheaper than optical cables and are less prone to compatibility and thermal issues, they are limited in length (2-3 m, up to 5 m maximum).

  • Active Optical Cables (AOCs): Although these cables are cheaper than dedicated transceivers and fiber optic cables, they might cause compatibility issues, or your NIC or switch might not support them.