2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter

Introduction

The 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter is a dual-port InfiniBand Host Channel Adapter (HCA) based on Mellanox ConnectX IB technology. When combined with a QDR switch, it delivers end-to-end 40 Gb bandwidth per port. This solution is suitable for low-latency, high-bandwidth, performance-driven server and storage clustering applications in High Performance Compute environments. The adapter uses the CFFh form factor and can be combined with CIOv or CFFv adapters for additional SAS, Fibre Channel, or Ethernet ports.

Figure 1: 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter

Description: A visual representation of the 2-Port 40 Gb InfiniBand Expansion Card (CFFh). The card features two InfiniBand ports on one end and a CFFh connector on the other. It has a heatsink and various electronic components.

Did you know?

InfiniBand is a scalable high-performance fabric used for Petascale computing. The Roadrunner supercomputer, one of the world's largest, utilized Mellanox ConnectX DDR adapters. QDR represents the next generation, offering double the bandwidth per port.

Part Number Information

Table 1 lists the part numbers for ordering the 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter.

The part number includes:

• One 2-Port 40 Gb InfiniBand Expansion Card (CFFh)
• Documentation package

Features and Specifications

The 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter includes the following features and specifications:

• Form factor: CFFh
• Host interface: PCI-E x8 Gen 2 (5.0GT/s) with 40+40 Gbps bidirectional bandwidth
• Dual 4X InfiniBand ports at speeds 10 Gbps, 20 Gbps, or 40 Gbps per port
• 6.5 GBps bidirectional performance
• RDMA, Send/Receive semantics
• Hardware-based congestion control
• Atomic operations
• 16 million I/O channels
• 256 to 4 Kb MTU
• 1 GB messages
• 9 virtual lanes: 8 data + 1 management
• 1µs MPI ping latency
• CPU offload of transport operations
• End-to-end QoS and congestion control
• TCP/UDP/IP stateless offload
• Enhanced InfiniBand features:
  • Hardware-based reliable transport
  • Hardware-based reliable multicast
  • Extended Reliable Connected transport
  • Enhanced Atomic operations
  • Fine grained end-to-end QoS
• Hardware-based I/O virtualization features:
  • Single Root IOV
  • Address translation and protection
  • Multiple queues per virtual machine
  • VMware NetQueue support
• Protocol support:
  • Open MPI, OSU MVAPICH, HP MPI, Intel MPI, MS MPI, Scali MPI
  • IPoIB, SDP, RDS
  • SRP, iSER, FCoIB and NFS RDMA

Operating Environment

The 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter supports the following environment:

• Temperature:
  • 10 to 52 °C (50 to 125.6 °F) at an altitude of 0 to 914 m (0 to 3,000 ft)
  • 10 to 49 °C (50 to 120.2 °F) at an altitude of 0 to 3000 m (0 to 10,000 ft)
• Relative humidity: 8% to 80% (noncondensing)

Supported Servers and I/O Modules

Table 2 lists the BladeCenter servers that the 2-Port 40 Gb InfiniBand Expansion Card for BladeCenter supports.

Figure 2: Location on the BladeCenter server planar where the CFFh card is installed

Description: A diagram illustrating the placement of the CFFh expansion card within a BladeCenter server. It highlights the CFFh connector, the expansion card itself, and high-speed connectors leading to I/O bays. A CIOv connector for further I/O expansion is also indicated.

BladeCenter chassis support is based on the blade server type. Consult IBM ServerProven for specific chassis compatibility: http://ibm.com/servers/eserver/serverproven/compat/us/.

Table 3 lists I/O modules supported with the 2-Port 40 Gb InfiniBand Expansion Card (CFFh). These modules are supported in BladeCenter H chassis only.

In BladeCenter H, the CFFh card ports connect through the midplane to I/O bays 7, 8, 9, and 10, as shown in Figure 3.

Figure 3: BladeCenter H I/O topology showing the I/O paths from CFFh expansion cards

Description: A diagram illustrating the I/O path within a BladeCenter H chassis. It shows a Blade Server with expansion cards (CFFh, CFFv) connecting to a Mid-Plane, which then routes connections to various I/O Bays (1 through 10). The diagram includes a legend for different connection types: Standard I/O bays, High-speed I/O bays, and Bridge modules I/O bays connections.

For each 4X InfiniBand port to be used, one I/O module must be installed. Table 4 details the specific I/O bays required. For the 2-Port 40 Gb InfiniBand Expansion Card (CFFh), install I/O modules in I/O bays 7/8 and 9/10 (two modules total, each occupying two adjacent high-speed slots).

Popular Configurations

Figure 4 illustrates a 40 Gb solution using the 2-Port 40 Gb InfiniBand Expansion Card (CFFh) and Voltaire 40 Gb InfiniBand Switch Modules. Two Voltaire 40 Gb InfiniBand Switch Modules are installed in bays 7/8 and 9/10 of the BladeCenter H chassis. Connections between expansion cards and switch modules are internal, requiring no external cabling.

Figure 4: A 40 Gb solution using 2-Port 40 Gb InfiniBand Expansion Card (CFFh) and Voltaire 40 Gb InfiniBand Switch Modules

Description: A diagram showing an IBM BladeCenter server and an IBM BladeCenter H chassis. It depicts how the 2-Port 40 Gb InfiniBand Expansion Card (CFFh) within the server connects to two Voltaire 40 Gb InfiniBand Switch Modules installed in the chassis's I/O bays (7/8 and 9/10).

Table 5 lists the components used in this configuration.

Operating System Support

The 2-Port 40 Gb InfiniBand Expansion Card (CFFh) supports the following operating systems:

• Microsoft Windows Server 2008 R2
• Microsoft Windows Server 2008, Datacenter x64 Edition
• Microsoft Windows Server 2008, Datacenter x86 Edition
• Microsoft Windows Server 2008, Enterprise x64 Edition
• Microsoft Windows Server 2008, Enterprise x86 Edition
• Microsoft Windows Server 2008, Standard x64 Edition
• Microsoft Windows Server 2008, Standard x86 Edition
• Microsoft Windows Server 2008, Web x64 Edition
• Microsoft Windows Server 2008, Web x86 Edition
• Red Hat Enterprise Linux 4 AS for AMD64/EM64T
• Red Hat Enterprise Linux 4 AS for x86
• Red Hat Enterprise Linux 5 Server Edition
• SUSE LINUX Enterprise Server 10 for AMD64/EM64T
• SUSE LINUX Enterprise Server 11 for AMD64/EM64T

Operating system support depends on the combination of the expansion card and the blade server. For the latest information on supported versions and service packs, consult IBM ServerProven: http://ibm.com/servers/eserver/serverproven/compat/us/. Select the blade server, then the expansion card to view supported operating systems.

Related Publications

For more information, refer to the following documents:

• 2-Port 40 Gb InfiniBand Expansion Card (CFFh) for BladeCenter Installation and User's Guide: http://www.ibm.com/support
• Voltaire 40 Gb InfiniBand Switch Module Installation Guide: http://www.ibm.com/support
• BladeCenter Interoperability Guide: http://lenovopress.com/bcig
• BladeCenter Products and Technology, SG24-7523: http://lenovopress.com/sg247523

Related Product Families

Product families related to this document include:

• Blade Network Adapters

Notices

Lenovo may not offer all products, services, or features discussed in this document in all countries. Consult your local Lenovo representative for current availability. Any reference to a Lenovo product, program, or service does not imply that only that specific item may be used; functionally equivalent alternatives that do not infringe Lenovo's intellectual property rights may be used. Users are responsible for evaluating and verifying the operation of any alternative product, program, or service. Lenovo may hold patents or pending patent applications related to this document. This document does not grant any license to these patents. License inquiries can be sent in writing to:

Lenovo (United States), Inc.
8001 Development Drive
Morrisville, NC 27560
U.S.A.
Attention: Lenovo Director of Licensing

LENOVO PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information may contain technical inaccuracies or typographical errors. Changes are made periodically and will be incorporated in new editions. Lenovo may make improvements and/or changes to the product(s) and/or program(s) described at any time without notice.

The products described are not intended for use in implantation or other life support applications where malfunction may result in injury or death. This document does not affect or change Lenovo product specifications or warranties. Nothing herein shall operate as an express or implied license or indemnity under Lenovo's or third parties' intellectual property rights. All information was obtained in specific environments and is presented as an illustration; results in other environments may vary. Lenovo may use or distribute submitted information without obligation.

References to non-Lenovo Web sites are for convenience only and do not constitute an endorsement. Use of those sites is at your own risk. Performance data was determined in a controlled environment; results in other environments may vary. Some measurements may have been made on development-level systems and may not be the same on generally available systems; some may be estimates through extrapolation. Actual results may vary. Users should verify applicable data for their specific environment.

© Copyright Lenovo 2022. All rights reserved.

This document, TIPS0700, was created or updated on December 7, 2010.

Send comments:

• Use the online Contact us review form: https://lenovopress.lenovo.com/TIPS0700
• Send comments via email to: comments@lenovopress.com

This document is available online at https://lenovopress.lenovo.com/TIPS0700.

Trademarks

Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States and/or other countries. A current list is available at https://www.lenovo.com/us/en/legal/copytrade/.

The following are trademarks of Lenovo in the United States and/or other countries:

• Lenovo®
• BladeCenter Interoperability Guide
• BladeCenter®
• ServerProven®

The following are trademarks of other companies:

• Intel® is a trademark of Intel Corporation or its subsidiaries.
• Linux® is the trademark of Linus Torvalds in the U.S. and other countries.
• Microsoft®, Windows Server®, and Windows® are trademarks of Microsoft Corporation in the United States and/or other countries.

Other company, product, or service names may be trademarks or service marks of others.