This section provides safety recommendations and guidance for identifying NICs and choosing transceivers and cables for your hardware platform.

This section provides safety recommendations and guidance for identifying NICs and choosing transceivers and cables for your hardware platform.

Safety Recommendations

  • Operating Ambient Temperature: When you install nodes in a closed or multi-unit rack assembly, the operating ambient temperature of the rack or cabinet environment can be greater than the ambient temperature of the room.

    Install your equipment in an environment where the maximum ambient temperature (Tma) doesn’t exceed 40° C (104° F).

  • Sufficient Airflow: Install your equipment in a way that avoids compromising the amount of airflow required for the equipment’s safe operation.

  • Even Mechanical Loading: Ensure your equipment is level when you mount it.

  • Circuit Load: Determine the load your equipment can place on your supply circuit and refer to nameplate ratings to determine potential circuit overload scenarios.

  • Reliable Earthing: Maintain earthing (by connecting to the part of the equipment that doesn’t carry current).

    Give particular attention to supply connections other than direct connections to the branch circuit (for example, power strips).

  • Redundant Power Supplies (PSU): When you receive multiple PSUs with your equipment, connect each PSU to a separate circuit.

  • Equipment Under Maintenance: Ensure that all equipment under maintenance has all of its PSUs disconnected.

  • Battery Replacement: Use compatible battery types to avoid a potential explosion. Dispose of depleted batteries according to provided instructions.

Identifying NICs and Choosing Transceivers and Cables

Step 1: Identify the NICs in Your Nodes

Most server platforms are compatible with multiple NIC models. The NIC model determines transceiver compatibility.

  1. Use SSH to connect to your node.

  2. Run the following command to list the Ethernet controllers:

    lspci | grep "Ethernet controller"

    A list of NICs appears. For example:

    45:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
45:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
81:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
81:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]

  3. To determine the speed and firmware compatibility information for the NICs in your node, refer to the following table.

Common NIC Models

NIC Model Speed Firmware Compatibility Information
82599ES 10 Gbps Intel 82599ES 10 Gigabit Ethernet Controller
ConnectX-3 10 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Firmware Release Notes
ConnectX-3 Pro 40 Gbps Supported Cables and Modules in the Mellanox ConnectX-3 Pro Firmware Release Notes
ConnectX-4 40 Gbps Firmware Compatible Products in the NVIDIA Mellanox ConnectX-4 Adapter Cards Firmware Release Notes
ConnectX-4 Lx 25 Gbps Firmware Compatible Products in the NVIDIA ConnectX-4 Lx Adapter Cards Firmware Release Notes
ConnectX-5 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes
ConnectX-6 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Adapter Cards Firmware Release Notes
ConnectX-6 Dx 100 Gbps Firmware Compatible Products in the NVIDIA ConnectX-6 Dx Adapter Cards Firmware Release Notes
E810-CQDA2 100 Gbps Intel Ethernet Network Adapter E810-2CQDA2
E810-XXVDA2 25 Gbps Intel Ethernet Network Adapter E810-XXVDA2
P2100G 100 Gbps Broadcom Optical Transceivers, Supported Cables for Broadcom Ethernet Network Adapters
P225P 25 Gbps Broadcom Optical Transceivers, Supported Cables for Broadcom Ethernet Network Adapters

For NICs not listed above, consult your hardware vendor’s documentation for transceiver compatibility information.

Step 2: Choose Transceivers for Your Nodes

This section lists and explains the differences between the types of transceivers available for your nodes.

  • Lucent Connector (LC): The LC with two fibers is very common for 10 Gbps and 25 Gbps connections.

  • Lucent Connector Duplex (LC Duplex): The LC duplex with two fibers is the most common standard for 25 Gbps connections. The maximum short-range connection is 100 m and long-range connection is 10 km. There is also an extended-range standard with a maximum of 40 km.

  • Multi-Fiber Push On (MPO): The MPO connector with eight fibers is a common connector for 40 Gbps connections.

  • PAM4: Some newer switches can establish 100 Gbps connections by using double 50 Gbps PAM4 connections instead of the more common four 25 Gbps connections. For information about configuring Pulse Amplitude Modulation 4-level (PAM4), see Auto-Negotiation on Ethernet NIC Controllers in the Broadcom documentation.

  • SR4: The SR4, with four QSFP28 connections over an eight-fiber cable, is the most common and cost-efficient standard for 100 Gbps connections. The maximum range for SR4 is 100 m.

Step 3: Choose Cables for Your Transceivers

This section lists and explains the differences between the types of cables available for your transceivers.

  • Optical Cables: We recommend using optical cables and optical transceivers that both the NIC and the switch support.

  • Direct Attach Cables (DACs): Although these cables are significantly cheaper than optical cables and are less prone to compatibility and thermal issues, they are limited in length (2-3 m, up to 5 m maximum).

  • Active Optical Cables (AOCs): Although these cables are cheaper than dedicated transceivers and fiber optic cables, they might cause compatibility issues, or your NIC or switch might not support them.