IBM POWER9 Enterprise E950: Unofficial Deep Dive

Introduction to POWER9 Enterprise Servers

The IBM POWER9 processor family introduces significant advancements for enterprise servers. This presentation focuses on the E950 model.

POWER9 Enterprise E980 Overview

• 192 POWER9 CPU cores @ 4.0 GHz
• 64 TB DDR4 CDIMM Memory
• 16 Internal NVMe for VIOS or OS booting
• 32 PCIe Gen4 adapters
• Approximately 30% performance jump from E880 due to whole system enhancements.

POWER9 Enterprise E950 Overview

• 48 POWER9 CPU cores @ 3.8 GHz
• 16 TB DDR4 ISDIMM Memory
• 4 Internal NVMe for VIOS or OS booting
• 10 PCIe Gen4 adapters + 1 Gen3
• Approximately 42% performance jump from E850.

POWER9 Server Performance Highlights:

• Super strength SMT threads
• CPU memory caching
• SMP bus bandwidth
• Memory sizes
• PCIe Gen4 boost

Availability and Upgrade Information

• Announcement Date: August 7th (E950 & E980)
• eConfig Date: August 8th
• General Availability Date: August 17th (E950)
• E980 Availability: Sept 21st (2 nodes = 96 core), Nov 16th (4 nodes = 192 core)
• Upgrade from POWER8 & POWER9 2- to 4-node MES upgrades are planned for Q4.

Before we start: Slides and replay available at http://tinyurl.com/PowerVUG. The presentation will share PowerPoint on the Power VUG website: http://tinyurl.com/AIXpert. It includes approximately 250 slides, with about 100 picture slides of a beta machine; GA servers may differ.

Document Notes and HMC Information

Disclaimer: The document may contain typos, mistakes, and evolving information. Updates can be found on the Power System VUG website. Please report any errors to Nigel Griffiths (nag@uk.ibm.com).

HMC News

• POWER9 requires HMC 920 software (an update to HMC 910).
• Supported on CR7, CR8, CR9, or HMC 7063-CR1 (POWER8 based).
• Offers more CPU, RAM, Disk, and slightly lower cost.
• Provides simple remote start/stop with ipmitool + WebUI.
• Gotchas:
• HMC Enhanced+ GUI (No Classic GUI).
• No POWER6 support.
• Older HMC hardware may not be compatible.
• IVM is not supported.
• Intel-based HMC may be out of stock in some countries.

POWER9 Processor Architecture

Processor Generation and Transistor Count

The transistor count has grown significantly across processor generations:

• POWER4 (2001): 0.2 billion transistors
• POWER5 (2004): 0.3 billion transistors
• POWER6 (2007): 0.8 billion transistors
• POWER7 (2010): 1.2 billion transistors
• POWER8 (2014): 4.2 billion transistors
• POWER9 (2017): 8.0 billion transistors

POWER9 Chip Variations

POWER9 offers different core designs optimized for various workloads:

• Slim Core (SMT4): Up to 22 CPU cores, optimized for Linux, HPC, and KVM. Found in models like AC922, LC922, LC921. Supports DDR4 RAM and Linux only.
• Fused Core (SMT8): Up to 12 CPU cores, optimized for PowerVM, throughput, and IBM Scale-Out. Supports DDR4 RAM and AIX + IBM i + Linux. Found in Enterprise E950 and E980 models.

The E950 does not support IBM i. Enterprise E980 supports up to 4 CECs, while E950 supports up to 2 CECs.

Core Threading

POWER9 cores feature advanced threading: SMT-4 Slim Cores and SMT-8 Fused Cores. Each CPU core thread can access multiple 'slices' of processing resources, with SMT-8 cores offering up to 8 slices per thread.

POWER9 Fused Core Strength vs. POWER8

POWER9 SMT8 cores offer significantly enhanced resources compared to POWER8 SMT8 cores, including more Vector Scalar Unit (VSU) pipes, Floating Point (FP) and Fixed Point (FXU) pipes, and Load Store Unit (LSU) slices, leading to higher performance.

Server Configurations and Adapters

Rack Options

The 7965-S42 is a 19-inch rack option. It is 42U high and 600mm wide. Other specifications include depth, rear door heat exchanger support, vertical pockets, and earthquake certification.

POWER9 Adapters

• PCIe GEN 4: Offers good future-proofing, doubles I/O rates, and supports next-generation adapters (40+ Gb/sec).
• Initially, most adapters will be GEN 3, providing sufficient speed and compatibility with GEN4 slots.
• Note: Not all adapters are supported at initial General Availability (GA).

Storage and Boot Options

POWER9 U.2 NVMe Support for E950

• The E950 features four external storage bays supporting NVMe drives.
• NVMe drives offer higher performance than SAS SSDs and support concurrent maintenance.
• They are primarily intended for storing and booting OS images (AIX/VIOS).
• Each NVMe device is a separate PCIe endpoint, assignable to different LPARs.
• Warranty is 5 years, with a 'fuel gauge' to monitor wear provided for AIX/Linux.

Internal DVD Support

The E950 does not have internal DVD support. External USB DVD drives or USB memory keys are recommended alternatives for OS installation.

NVMe Storage Options

Available NVMe drive sizes include 800 GB, 1.6 TB, and 3.2 TB. These are ideal for VIOS boot or OS boot devices, offering faster performance and 2.4 drive write per day DWPD (5-year warranty).

Backplane Options

Three backplane options are available, depending on SAS internal disk support:

• Zero DASD Backplane: Supports only NVMe drives, no SAS adapter.
• Base DASD Backplane: Includes 1 SAS PCIe adapter with selected SAS drives.
• Split DASD Backplane: Features 2 SAS PCIe adapters, splitting disks between them (4+4 configuration).

Server Range and Models

POWER8 Range Overview

The POWER8 server range included models like E870, E850, and various S-series rackmount and blade servers, offering configurations from 16-48 cores up to 8-192 cores with varying memory and I/O capabilities.

POWER9 Range in 2018

• Scale-Out (Q2): S922 (2U/4U, 1-2 socket, 4-24 cores, 4TB RAM) and S924 (4U, 2-4 socket, 16-48 cores, 16TB RAM).
• Midrange (Q3): E950 (4U, 4 socket, 16-48 cores, 16TB RAM).
• Enterprise (Q3+Q4): E980 (7U-22U, 4-16 socket, 8-192 cores, 16TB RAM).
• Other: AC922 (Q4 2017), LC922/LC921 (Q2).

IBM E950 System Specifications

System Configuration (9040-MR9) - 4U

• Processors: Two or four POWER9 processors, supporting 16, 20, 22, 24, 32, 40, 44, 48 CPU cores per processor. Maximum clock speed is 3.8 GHz.
• Sockets: 2 or 4, field upgradable.
• Memory: Up to 16 TB total DDR4 ISDIMMs across 8 riser cards (128 DIMM slots total). Supports 8, 16, 32, 64, 128 GB DIMM sizes.
• Media Bays: External USB DVD or USB flash key for OS installation.
• Integrated PCIe: Full Height slots including PCIe Gen 4 (eight x16 + two x8 for 2 processors) and PCIe Gen 3 (one x8 for default Ethernet).
• Internal SAS Bays: Up to 8 SAS 2.5-inch drives (15mm) via 1 or 2 SAS adapters. Supports JBOD, RAID 1, 5, 6, or 10. Also supports 1 to 4 NVMe U.2 drives (2.5-inch 7mm).
• Max I/O Drawers: Supports EMX I/O drawers (EXP12SX, ESP24SX, EXP24S).
• External Storage Drawers: Supports EXP12SX, ESP24SX, EXP24S, requiring a PCIe SAS adapter or SAS port.
• Software Stack: HMC 920+, PowerVM 2.2.6.23+, AIX (6.1, 7.1, 7.2), Linux (SLES, RHEL). IBM i is not supported.
• Warranty: 3 Years 24x7.

E950 Front and Rear Views

The front view shows the operator panel LCD, USB 3.0 ports, system fans, and bays for SAS and NVMe drives. The rear view displays USB 3.0 ports, power supplies, and PCIe cassette connections.

Processor Architecture Highlights

The E950 features four directly connected POWER9 processors, forming a 1-hop interconnect for GPU & CAPI, RAM, I/O, and SMP communications. It utilizes PCIe Gen4 for high-speed I/O.

E950 Processor Options and Performance

Processor configurations range from 8 to 12 cores per socket, scaling up to 48 cores in a 4-socket system. Performance is measured by rPerf, with higher core counts generally yielding higher throughput and better rPerf per core.

• E950 makes sense if: you plan to upgrade later or need 8 TB memory (S924 max is 4TB).

Server Generation Performance (rPerf for E950 48 CPU cores)

The E950 shows a significant performance increase over previous generations:

• POWER5: 306
• POWER6: 430 (+167%)
• POWER7: 517 (+121%)
• POWER7+: 613 (+87%)
• POWER8: 747 (+53%)
• POWER8C: 810
• POWER9 E950: 1146 (+42% over E850C)

Memory Subsystem

POWER9 Processor Family Memory Architectures

Two main architectures exist: 'Scale Out' (Direct Attach Memory with 8 DDR4 Ports, typically 2-socket systems) and 'Scale Up' (Buffered Memory with 8 Buffered Channels, typically 4 to 16-socket systems). The E950 uses the Scale Up architecture.

Memory Riser Card

Each memory riser card supports 16 DDR4 DIMMs and features 4 Centaur L4 Cache chips. A minimum of 8 white slots is required on a riser. The system can accommodate up to 8 riser cards, totaling 16 TB of memory.

E950 Memory Subsystem Highlights

• DIMM Sizes: 8, 16, 32, 64, 128 GB DDR4 DIMMs.
• Per E950 Server: Max 920 GB/s memory bandwidth.
• Per POWER9 Processor: Max 230 GB/s memory bandwidth.
• Memory Capacity: Up to 16 TB total memory. Minimum configuration for 2-socket is 64 GB, for 4-socket is 128 GB.
• Memory Increment: DIMMs are added in packs of eight.

Memory Rules (min/max for 2 or 4 procs): All memory must be DDR4 and the same speed. For 2-socket systems, min 64 GB, max 8 TB. For 4-socket systems, min 128 GB, max 16 TB. It is recommended not to go below 256 GB or 512 GB for optimal performance.

I/O and Connectivity

E950 I/O Subsystem Organization

The E950's I/O subsystem is designed for high bandwidth and connectivity. Diagrams illustrate the arrangement of processors, memory risers, PCIe Gen4 and Gen3 slots, NVMe drives, and external ports, showing how components are interconnected.

Internal SAS Cabling

Various internal SAS cabling configurations are supported, including:

• All 8 disks on a single SAS adapter (EJ0K).
• Split disk configurations (4+4) using dual SAS adapters.
• High-performance configurations utilizing large shared adapter disk cache with 2 x EJ14 adapters.

Dual VIOS Suggestions

• Network: Recommend 2 or 4 redundant Ethernet adapters.
• Storage:
• 1 to 4 NVMe drives for fast, low-downtime storage.
• 2 or 4 Fibre Channel SAN connections for redundant SAN booting.
• Internal disks/SSD via two SAS adapters (split backplane) for mirrored storage.
• Remote Disk Drawer options are also available.

Internal Storage Numbering

Diagrams show the front bay layout, indicating the numbering for SAS disks and NVMe drives, and their assignment to SAS adapters for VIOS A and VIOS B.

POWER9 I/O Planned for GA1 and August 17th Release

Details are provided on planned I/O adapters for GA1 and the August 17th release, categorized by function (Network, Disk, CPU+RAM) and including FC, CCIN, Description, and Max OS Support (AIX/Linux).

• Network Adapters: Include Fibre Channel adapters (8Gb, 16Gb, 32Gb) and Ethernet adapters (1GbE, 10Gb, 100Gb ROCE).
• Disk Adapters: Feature SAS adapters (RAID, Tape/DVD, Cache RAID), NVMe SSDs (800GB, 1.6TB, 3.2TB), and various SAS disk drives (SFF-2, SFF-3, LFF-1) with different RPMs and capacities.
• Memory Options: DDR4 DIMMs (8GB to 128GB) and Processor options (8-core to 12-core).

Operating System Support

Operating Systems for Enterprise

The E950 supports key enterprise operating systems:

• AIX: Supported with various levels, including AIX 7.2 TL2+SP2 and TL3. AIX with POWER9 support allows LPM from P7/P8 to POWER9.
• Linux: SUSE SLES and Red Hat RHEL are supported. Native Ubuntu is fully supported on base metal and for KVM hosting (OPAL mode), but Ubuntu on PowerVM is no longer supported by IBM.
• IBM i: Not supported on the E950.
• VIOS: Version 2.2.6.23 or later is recommended. VIOS 3.1 is expected soon.

Physical and Power Specifications

POWER9 E950 Power Supply

• Features concurrent maintenance and redundant power with four 2000 W (200-240 VAC) units.
• Achieves 80+ Platinum and EPA Energy Star compliance for energy efficiency.
• Includes advanced thermal and power management.

Physical Dimensions and Weight

• Size: 4U Rack Enclosure (17.5 in. width, 6.9 in. height, 35.6 in. depth). It is 5 inches greater in depth than the E850.
• Weight: Approximately 69 kg (152 lb), or ~121 lb with weight reduction. A four-person lift or lifting tool is recommended.

E950 Electrical Power

• Operating Voltage: 200 - 240 V AC.
• Frequency: 50 - 60 Hz +/- 3 Hz.
• Power Consumption: 3,850 watts maximum.
• Power Source Loading: 3.9 kVA maximum.
• The Model 9040-MR9 uses four power supply units with C19/C20 connectors for higher amps.

Thermal Output and Noise Level

• Thermal Output: 14,403 Btu/hr maximum per system node.
• Maximum Altitude: 3,050 m (10,000 ft).
• Noise Level: Approximately 7.4 bels (operating/idle) for a typical 8/12-core, 2TB memory configuration, and 8.1 bels (heavy workload) for a maximum configured system.

Call to Action

Good News

• The E950 offers significant improvements over the E850.
• Supports up to 48 CPU cores with approximately 42% POWER9 performance boost.
• Memory capacity increased from 4 TB to 16 TB.
• Easy-to-use rear PCIe cassette access for adapters.
• HMC readiness is recommended.

Challenges

• Electricity: C19/C20 connectors may require PDU changes due to higher amps.
• Size: The increased depth requires checking racks and doors.
• Heavy: Loading into racks requires care and a lifting device.
• Noisy: Ear protection may be needed depending on computer room policy.

Power E950 Specification Highlights

• 4U Server, 19" Rack Enclosure.
• 2 or 4 POWER9 Enterprise SMT8 processors enabling 1-Hop processor fabric interconnect.
• Up to 16TB Total DDR4 DIMMs (up to 4TB per processor).
• 920 GB/s memory bandwidth per System, 230 GB/s per Processor.
• 128 DIMM slots across 8 memory riser cards (16 DIMM slots per riser).
• Capacity on Demand for Processor and Memory.
• 10 PCIe Gen4 slots, 1 PCIe Gen3 slot; supports blind swap, full height, half length.
• Four High Speed 25Gb/s acceleration ports for future accelerators.
• 4 x NVMe Flash U.2 Bays (bootable).
• 8 x internal 8 SFF (2.5") SAS bays.
• Storage controller adapters plug into PCIe slots.
• Supports single or split backplanes for SAS storage PCIe adapters.
• SAS PCIe adapters are concurrently maintainable.
• Enhanced DC-DC Regulator Redundancy.
• Full Fan Concurrent Maintenance.
• Supports I/O Expansion and Storage drawers.
• No Charge: PowerVM & PowerVC.