ALPHA DATA ADM-PCIE-9H3 High Performance FPGA Processing Card

Introduction

The ADM-PCIE-9H3 is a high-performance reconfigurable computing card intended for Data Center applications, featuring a Xilinx Virtex UltraScale+ Plus FPGA with High Bandwidth Memory (HBM).

Key Features

• PCIe Gen1/2/3 x1/2/4/8/16 capable
• Passive and active thermal management configuration
• 1/2 length, low profile, x16 edge PCIe form factor
• 8GB HBM on-die memory capable of 460GB/s
• One QSFP-DD cage capable of data rates up to 28 Gbps per 8 channels (224 Gbps)
• One 8 lane Ultraport SlimSAS connectors compliant with OpenCAPI and suitable for IO expansion
• Supports either VU33P or VU35P Virtex UltraScale+ FPGAs
• Front panel and rear edge JTAG access via USB port
• FPGA configurable over USB/JTAG and SPI configuration flash
• Voltage, current, and temperature monitoring
• 8 GPIO signals and 1 isolated timing input

Order Code
ADM-PCIE-9H3
ADM-PCIE-9H3/NF (without optional fan)
See http://www.alpha-data.com/pdfs/adm-pcie-9h3.pdf for complete ordering options.

Board Information

Physical Specifications
The ADM-PCIE-9H3 complies with PCI Express CEM revision 3.0.
Table 1 : Mechanical Dimensions (Inc. Front Panel)

Chassis Requirements

PCI Express
The ADM-PCIE-9H3 is capable of PCIe Gen 1/2/3 with 1/2/4/8/16 lanes, using the Xilinx Integrated Block for PCI Express.

Mechanical Requirements
A 16-lane physical PCIe slot is required for mechanical compatibility.

Power Requirements
The ADM-PCIE-9H3 draws all power from the PCIe Edge. As per PCIe specification, this limits the power consumption of the card to a maximum 75W.
Power consumption estimation requires the use of the Xilinx XPE spreadsheet and a power estimator tool available from Alpha Data. Please contact support@alpha-data.com to obtain this tool.
The power available to the rails calculated using XPE are as follows:

Table 2 : Available Power By Rail

Thermal Performance
If the FPGA core temperature exceeds 105 degrees Celsius, the FPGA design will be cleared to prevent the card from over-heating.
The ADM-PCIE-9H3 comes with a heat sink to reduce the temperature of the FPGA, which is typically the hottest point on the card. The FPGA die temperature must remain under 100 degrees Celsius. To calculate the FPGA die temperature, take your application power, multiply by Theta JA from the table below, and add to your system internal ambient temperature. The graph below shows two lines, one was tested in a duct with the shrouds installed, and the other was tested without the shrouds. The performance is generally better without the shrouds, but they do provide improved handling and reduce air re-circulation in compact servers. The shroud can be removed using a 1/16″ hex driver. If you are using the fan provided with the board, you will find theta JA is approximately 1.43 degC/W for the board in still air with or without the shroud installed.
The power dissipation can be estimated by using the Alpha Data power estimator in conjunction with the Xilinx Power Estimator (XPE) downloadable at http://www.xilinx.com/products/technology/power/xpe.html. Download
the UltraScale tool and set the device to Virtex UltraScale+, VU33P, FSVH2104, -2, -2L, or -3, extended. Set the ambient temperature to your system ambient and select ‘user override’ for the effective theta JA and enter the figure associated with your system LFM in the blank field. Proceed to enter all applicable design elements and utilization in the following spreadsheet tabs. Next aquire the 9H3 power estimator from Alpha Data by contacting
support@alpha-data.com. You will then plug in the FPGA power figures along with Optical module figures to get a board level estimate.

Active VS Passive Thermal Management
The ADM-PCIE-9H3 ships with a small optional blower for active cooling in systems with poor airflow. If the ADM-PCIE-9H3 will be installed in a server with controlled airflow, the order option /NF can be used to receive cards without this extra piece. The fans have a much shorter mean time between failure (MTBF) than the rest of the assembly, so passive cards have much longer life expectance before requiring maintenance. The ADM-PCIE-9H3 also includes a fan speed controller, allowing variable fan speed based on die temperature, and
detection of a failed fan (see section Fan Controllers).

Customizations
Alpha Data provides extensive customization options to existing commercial off-the-shelf (COTS) products.
Some options include, but are not limited to: additional networking cages in adjacent slots or full profile, enhanced heat sinks, baffles, and circuit additions.
Please contact sales@alpha-data.com to get a quote and start your project today.

Functional Description

Overview
The ADM-PCIE-9H3 is a versatile reconfigurable computing platform with a Virtex UltraScale+ VU33P/VU35P FPGA, a Gen3x16 PCIe interface, 8GB of HBM memory, one QSFP-DD cage, an OpenCAPI compatible Ultraport SlimSAS connector also capable of 28G/channel, an isolated input for a timing synchronization pulse, a 12 pin header for general purpose use (clocking, control pins, debug, etc.), front panel LEDs, and a robust system monitor.

Switches
The ADM-PCIE-9H3 has an octal DIP switch SW1, located on the rear side of the board. The function of each switch in SW1 is detailed below:

Table 3 : Switch Functions

Use IO Standard “LVCMOS18” when constraining the user switch pins.

LEDs
There are 7 LEDs on the ADM-PCIE-9H3, 4 of which are general purpose and whose meaning can be defined by the user. The other 3 have fixed functions described below:

Table 4 : LED Details

See Section Complete Pinout Table for full list of user controlled LED nets and pins

Clocking
The ADM-PCIE-9H3 provides flexible reference clock solutions for the many multi-gigabit transceiver quads and FPGA fabric. Any clock out of the Si5338 Clock Synthesizer is re-configurable from either the front panel USB USB Interface or the Alpha Data sysmon FPGA serial port. This allows the user to configure almost any arbitrary clock frequency during application run time. Maximum clock frequency is 312.5MHz.
There is also an available Si5328 jitter attenuator. This can provide clean and synchronous clocks to the QSFP-DD and OpenCAPI (SlimSAS) quad locations at many clock frequencies. These devices only use volatile memory, so the FPGA design will need to re-configure the register map after any power cycle event.
All clock names in the section below can be found in Complete Pinout Table.

Si5328
If jitter attenuation is required please see the reference documentation for the Si5328.
https://www.silabs.com/Support%20Documents/TechnicalDocs/Si5328.pdf
The circuit connections mirror Xilinx VCU110 and VCU108, please see Xilinx Dev Boards for references

PCIe Reference Clocks
The 16 MGT lanes connected to the PCIe card edge use MGT tiles 224 through 227 and use the system 100 MHz clock (net name PCIE_REFCLK).
Alternatively, a clean, onboard 100MHz clock is available as well (net name PCIE_LCL_REFCLK).

Fabric Clock
The design offers a fabric clock (net name FABRIC_SRC_CLK) which defaults to 300 MHz. This clock is intended to be used for IDELAY elements in FPGA designs. The fabric clock is connected to a Global Clock (GC) pin.
DIFF_TERM_ADV = TERM_100 is required for LVDS termination

Auxiliary Clock
The design offers a auxiliary clock (net name AUX_CLK) which defaults to 300 MHz. This clock can be used for any purpose and is connected to a Global Clock (GC) pin.
DIFF_TERM_ADV = TERM_100 is required for LVDS termination

Programming Clock (EMCCLK)
A 100MHz clock (net name EMCCLK_B) is fed into the EMCCLK pin to drive the SPI flash device during configuration of the FPGA. Note that this is not a global clock capable IO pin.

QSFP-DD
The QSFP-DD cage is located in MGT tiles 126 and 127 and use a 161.1328125MHz default reference clock.
Note that this clock frequency can be changed to any arbitrary clock frequency up to 312MHz by re-programing the Si5338 reprogrammable clock oscillator via system monitor. This can be done using the Alpha Data API or over USB with the appropriate Alpha Data Software tools.
See net names QSFP_CLK* for pin locations.
The QSFP-DD cage is also located such that it can be clocked from the Si5328 jitter attenuator clock multiplier.
See net names SI5328_OUT_1* for pin locations.

Ultraport SlimSAS (OpenCAPI)
The Ultraport SlimSAS connector is located in MGT tile 124 and 125.
For OpenCAPI an external 156.25MHz clock is provided over the cable. See net names CAPI_CLK_0* for cable clock pin locations.
Another alternative clock source for this interface is the Si5338 clock synthesizer which is defaulted to 161.1328125MHz. See net names CAPI_CLK_1* for pin locations. Note that this clock frequency can be changed to any arbitrary clock frequency up to 312MHz by re-programing the Si5338 reprogrammable clock oscillator via system monitor. This can be done using the Alpha Data API or over USB with the appropriate Alpha Data Software tools.
For jitter sensitive applications, this interface can be clocked from the Si5328 jitter attenuator. See net names SI5328_OUT_0* for pin locations.

PCI Express

The ADM-PCIE-9H3 is capable of PCIe Gen 1/2/3 with 1/2/4/8/16 lanes. The FPGA drives these lanes directly using the Integrated PCI Express block from Xilinx. Negotiation of PCIe link speed and number of lanes used is generally automatic and does not require user intervention.
PCI Express reset (PERST#) connected to the FPGA at two location. See Complete Pinout Table signals PERST0_1V8_L and PERST1_1V8_L.
The other pin assignments for the high speed lanes are provided in the pinout attached to the Complete Pinout Table
The PCI Express specification requires that all add-in cards be ready for enumeration within 120ms after power is valid (100ms after power is valid + 20ms after PERST is released). The ADM-PCIE-9H3 does meet this requirement when configured from a tandem bitstream with the proper SPI constraints detailed in the section:
Configuration From Flash Memory. For more details on tandem configuration, see Xilinx xapp 1179.

Note:
Different motherboards/backplanes will benefit from different RX equalization schemes within the PCIe IP core provided by Xilinx. Alpha Data recommends using the following setting if a user experiences link errors or training issues with their system: within the IP core generator, change the mode to “Advanced” and open the “GT Settings” tab, change the “form factor driven insertion loss adjustment” from “Add-in Card” to “Chip-to-Chip” (See Xilinx PG239 for more details).

QSFP-DD
One QSFP-DD cage is available at the front panel. This cage is capable of housing either QSFP28 or QSFP-DD cables (backwards compatible). Both active optical and passive copper QSFP-DD/QSFP28 compatible models are fully compliant. The communication interface can run at up to 28Gbps per channel. There are 8 channels across the QSFP-DD cage (total maximum bandwidth of 224Gbps). This cage is ideally suited for 8x 10G/25G, 2x 100G Ethernet, or any other protocol supported by the Xilinx GTY Transceivers. Please see Xilinx User Guide UG578 for more details on the capabilities of the transceivers.
The QSFP-DD cage has control signals connected to the FPGA. The connectivity is detailed in the Complete Pinout Table at the end of this document. The notation used in the pin assignments is QSFP* with locations clarified in the diagram below.
Use the QSFP_SCL_1V8 and QSFP_SDA_1V8 pins as detailed in Complete Pinout Table to communicate with QSFP28 register space.

Note:
The LP_MODE (Low Power Mode) to the cage is tied to ground, use the management interface to set power rules.

It is possible for Alpha Data to pre-fit the ADM-PCIE-9H3 with QSFP-DD and QSFP28 components. The table below shows the part number for the transceivers fitted when ordered with this board.
Table 5 : QSFP28 Part Numbers

OpenCAPI Ultraport SlimSAS

An Ultraport SlimSAS receptacles along the back of the board allow for OpenCAPI compliant interfaces running at 200G (8 channels at 25G). Please contact support@alpha-data.com or your IBM representative for more details on OpenCAPI and its benefits.
The SlimSAS connector can also be used to connect an additional 2x QSFP28 breakout board, contact sales@alpha-data.com for more details. Alternatively, cabling cab be used to connect multiple ADM-PCIE-9H3 cards within a chassis.

System Monitor
The ADM-PCIE-9H3 has the ability to monitor temperature, voltage, and current of the system to check on the operation of the board. The monitoring is implemented using an Atmel AVR microcontroller.
If the core FPGA temperature exceeds 105 degrees Celsius, the FPGA will be cleared to prevent damage to the card.
Control algorithms within the microcontroller automatically check line voltages and on board temperatures and shares makes the information available to the FPGA over a dedicated serial interface built into the Alpha Data reference design package (sold separately). The information can also be accessed directly from the microcontroller over the USB interface on the front panel or via the IPMI interface available at the PCIe card edge.

Table 6 : Voltage, Current, and Temperature Monitors

System Monitor Status LEDs
LEDs D5 (Red) and D6 (Green) indicate the card health status.

Table 7 : Status LED Definitions

Fan Controllers
The onboard USB bus controlled by the system monitor has access to a MAX6620 fan controller. This device can be controlled through the multiple onboard system monitor communication interfaces, including USB, PCIe Edge SMBUS, and FPGA sysmon seral communication port. The fan controller is on I2C bus 1 at address 0x2a. For additional questions. Contact support@alpha-data.com with additional questions on utilizing these controllers.

USB Interface
The FPGA can be configured directly from the USB connection on either the front panel or the rear card edge.
The ADM-PCIE-9H3 utilizes the Digilent USB-JTAG converter box which is supported by the Xilinx software tool suite. Simply connect a micro-USB AB type cable between the ADM-PCIE-9H3 USB port and a host computer with Vivado installed. Vivado Hardware Manager will automatically recognize the FPGA and allow you to configure the FPGA and the SBPI configuration PROM.
The same USB connector is used to directly access the system monitor system. All voltages, currents, temperatures, and non-volatile clock configuration settings can be accessed using Alpha Data’s avr2util software at this interface.
Avr2util for Windows and the associated USB driver is downloadable here:
https://support.alpha-data.com/pub/firmware/utilities/windows/
Avr2util for Linux is downloadable here:
https://support.alpha-data.com/pub/firmware/utilities/linux/
Use “avr2util.exe /?” to see all options.
For example “avr2util.exe /usbcom com4 display-sensors” will display all sensor values.
For example “avr2util.exe /usbcom com4 setclknv 1 156250000” will set the QSFP clock to 156.25MHz. setclk index 0 = CAPI_CLK_1, index 1 = QSFP_CLK, index 2 = AUX_CLK, index 3 = FABRIC_CLK.
Change ‘com4’ to match the com port number assigned under windows device manager

Configuration
There are two main ways of configuring the FPGA on the ADM-PCIE-9H3:

• From Flash memory, at power-on, as described in Section 3.8.1
• Using USB cable connected at either USB port Section 3.8.2

Configuration From Flash Memory
The FPGA can be automatically configured at power-on from two 256 Mbit QSPI flash memory device configured as an x8 SPI device (Micron part numbers MT25QU256ABA8E12-0). These flash devices are typically divided into two regions of 32 MiByte each, where each region is sufficiently large to hold an uncompressed bitstream for a VU33P FPGA.
The ADM-PCIE-9H3 is shipped with a simple PCIe endpoint bitstream containing a basic Alpha Data ADXDMA bitstream. Alpha Data can load in other custom bitstreams during production test, please contact sales@alpha-data.com for more details.
It is possible to use Multiboot with a fallback image on this hardware. The master SPI configuration interface and the Fallback MultiBoot are discussed in detail in Xilinx UG570. At power-on, the FPGA attempts to configure itself automatically in serial master mode based on the contents of the header in the programing file. Multibook and ICAP can be used to selected between the two configuration regions to be loaded into the FPGA. See Xilinx UG570 MultiBoot for details.
The image loaded can also support tandem PROM or tandem PCIE with field update configuration methods.
These options reduce power-on load times to help meet the PCIe reset timing requirements. Tandem with field also enables a host system to reconfigure the user FPGA logic without losing the PCIe link, a useful feature when system resets and power cycles are not an option.
The Alpha Data System Monitor is also capable of reconfiguring the flash memory and reprograming the FPGA.
This provides a useful failsafe mechanism to re-program the FPGA even if it drops off the PCIe bus. The system monitor can be accessed over USB at the front panel and rear edge, or over the SMBUS connections on the PCIe edge.

Building and Programming Configuration Images

Generate a bitfile with these constraints (see xapp1233):

• set_property BITSTREAM.GENERAL.COMPRESS TRUE [ current_design ]
• set_property BITSTREAM.CONFIG.EXTMASTERCCLK_EN {DIV-1} [current_design]
• set_property BITSTREAM.CONFIG.SPI_32BIT_ADDR YES [current_design]
• set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 8 [current_design]
• set_property BITSTREAM.CONFIG.SPI_FALL_EDGE YES [current_design]
• set_property BITSTREAM.CONFIG.UNUSEDPIN {Pullnone} [current_design]
• set_property CFGBVS GND [ current_design ]
• set_property CONFIG_VOLTAGE 1.8 [ current_design ]
• set_property BITSTREAM.CONFIG.OVERTEMPSHUTDOWN Enable [current_design]

Generate an MCS file with these properties (write_cfgmem):

• -format MCS
• -size 64
• -interface SPIx8
• -loadbit “up 0x0000000 <directory/to/file/filename.bit>” (0th location)
• -loadbit “up 0x2000000 <directory/to/file/filename.bit>” (1st location, optional)

Program with vivado hardware manager with these settings (see xapp1233):

• SPI part: mt25qu256-spi-x1_x2_x4_x8
• State of non-config mem I/O pins: Pull-none
• Target the four files generated from the write_cfgmem tcl command.

Configuration via JTAG
A micro-USB AB Cable may be attached to the front panel or rear edge USB port. This permits the FPGA to be reconfigured using the Xilinx Vivado Hardware Manager via the integrated Digilent JTAG converter box. The device will be automatically recognized in Vivado Hardware Manager.
For more detailed instructions, please see “Using a Vivado Hardware Manager to Program an FPGA Device” section of Xilinx UG908: https://www.xilinx.com/support/documentation/sw_manuals/xilinx2014_1/ug908-vivado-programming-debugging.pdf

GPIO Connector
The GPIO option consists of a versatile shrouded connector from Molex with part number 87832-1222 that give users with custom IO requirements four direct connect to FPGA signals.
Recommended mating plug: Molex 0875681273 or 0511101260

Direct Connect FPGA Signals
8 nets are broken out to the GPIO header, as four sets of differential pairs. These signal are suitable for any 1.8V supported signaling standards supported by the Xilinx UltraScale architecture. See Xilinx UG571 for IO options.
LVDS and 1.8 CMOS are popular options. The 0th GPIO signal index is suitable for a global clock connection.
The direct connect GPIO signals are limited to 1.8V by a quickswitch (74CBTLVD3245PW) in order to protect the FPGA from overvoltage on IO pins. This quickswitch allows the signals to travel in either direction with only 4 ohms of series impedance and less than 1ns of propagation delay. The nets are directly connected to the FPGA after the quickswitch.
Direct connect signal names are labeled GPIO_0_1V8_P/N and GPIO_1_1V8_P/N, etc. to show polarity and grouping. The signal pin allocations can be found in Complete Pinout Table

Timing Input
J1.1 and J1.2 can be used as an isolated timing input signal (up to 25MHz). Applications can either directly connect to the GPIO connector, or Alpha Data can provide a cabled solution with an SMA or similar connector on the front panel. Contact sales@alpha-data.com for front panel connector options.
For pin locations, see signal name ISO_CLK in Complete Pinout Table.
The signal is isolated through a optical isolator part number TLP2367 with a 220 ohm of series resistance.

User EEPROM
A 2Kb I2C user EEPROM is provided for storing MAC addresses or other user information. The EEPROM is part number CAT34C02HU4IGT4A
The address pins A2, A1, and A0 are all strapped to a logical ‘0’.
Write protect (WP), Serial Clock (SCL), and Serial Data (SDA) pin assignments can be found in Complete Pinout Table with the names SPARE_WP, SPARE_SCL, and SPARE_SDA respectively.
WP, SDA, and SCL signals all have external pull-up resistors on the card.

Appendix A: Complete Pinout Table

Table 8 : Complete Pinout Table (continued on next page)

Revision History

Customer Service

© 2022 Copyright Alpha Data Parallel Systems Ltd.
All rights reserved.
This publication is protected by Copyright Law, with all rights reserved. No part of this publication may be reproduced, in any shape or form, without prior written consent from Alpha Data Parallel Systems Ltd.
Head Office
Address: Suite L4A, 160 Dundee Street,
Edinburgh, EH11 1DQ, UK
Telephone: +44 131 558 2600
Fax: +44 131 558 2700
email: sales@alpha-data.com
website: http://www.alpha-data.com
US Office
Address: 10822 West Toller Drive, Suite 250
Littleton, CO 80127
Telephone: (303) 954 8768
Fax: (866) 820 9956 – toll free
email: sales@alpha-data.com
website: http://www.alpha-data.com

All trademarks are the property of their respective owners.
Address: Suite L4A, 160 Dundee Street,
Edinburgh, EH11 1DQ, UK
Telephone: +44 131 558 2600
Fax: +44 131 558 2700
email: sales@alpha-data.com
website: http://www.alpha-data.com
Address: 10822 West Toller Drive, Suite 250
Littleton, CO 80127
Telephone: (303) 954 8768
Fax: (866) 820 9956 – toll free
email: sales@alpha-data.com
website: http://www.alpha-data.com

Documents / Resources

	ALPHA DATA ADM-PCIE-9H3 High Performance FPGA Processing Card [pdf] User Manual ADM-PCIE-9H3 High Performance FPGA Processing Card, ADM-PCIE-9H3, High Performance FPGA Processing Card, FPGA Processing Card, Processing Card
	ALPHA DATA ADM-PCIE-9H3 High Performance FPGA Processing Card [pdf] User Manual ADM-PCIE-9H3 High Performance FPGA Processing Card, ADM-PCIE-9H3, High Performance FPGA Processing Card, Performance FPGA Processing Card, FPGA Processing Card, Processing Card

References

• User Manual