Contents at a Glance
Contents
About the Authors
About the Technical Reviewers
Contributing Authors
Acknowledgments
Chapter 1: Why Data Center Efficiency Matters
	An Industry’s Call to Action
		Data Center Infrastructure Energy Use
		Energy Proportional Server Efficiency
		Regulatory Environment
	Measuring Energy Efficiency
		SPECPower
	High Performance Computing Efficiency
	Energy Efficiency and Cost
	Summary
Chapter 2: CPU Power Management
	Server CPU Architecture/Design
		CPU Architecture Building Blocks
			Threads, Cores, and Modules
			Caches and the Cache Hierarchy
			Dies and Packages
			On-die Fabrics and the Uncore
			Power Control Unit
			External Communication
			Thermal Design
		CPU Design Building Blocks
			Digital Synchronous Logic and Clocks
			SRAM and eDRAM
			I/O
		Intel Server Processors
	Introduction to Power
		CPU Power Breakdown
			Logic Power
			I/O Power
		Frequency, Voltage, and Temperature Interactions
	Power-Saving Techniques
		Turn It Off
		Turn It Down
		Power-Saving Strategies
			Race to Idle vs. Slow Down
	CPU Power and Performance States
		C-States
			Thread C-States
			Core C-States
				Core C0
				Core C1 and C1e
				Core C3
				Core C6
				Core C7 (and up)
				C-State Demotion
			Package C-States
			Module C-States
		P-States
			Per Socket P-States
			Per Core P-States
			Uncore Frequency Scaling
			Turbo
				Turbo Architecture
				Power/Thermal Limits
				Thermal Protection
				Electrical Protection
			C-States and Turbo
			Fused Turbo Frequencies
		T-States
		S-States and G-States
		S0i x
		Running Average Power Limit (RAPL)
			IMON and Digital Power Meter
			Linpack Example
			DRAM (Memory) RAPL
		CPU Thermal Management
			Prochot
		CPU Power Management Summary
	Summary
Chapter 3: Memory and I/O Power Management
	System Memory
		Memory Architecture Basics
		Devices and Ranks
		Memory Error Correction (ECC)
		Memory Capacity
		Device Power Characteristics
		DDR3 vs. DDR4
		RDIMMs, UDIMMs, SODIMMs, and LRDIMMs
		Memory Channel Interleave and Imbalanced Memory Configurations
	Power and Performance States
		CKE Power Savings
		Self-Refresh
		Voltage/Frequency
	DDR Thermal Management
		Monitoring Temperature
		Memory Throttling
	CPU DDRIO
	Workload Behavior
	Memory Reliability Features
	CPU I/Os
		CPU Interconnect
			Link Power States
		PCIe
			Link Power States
			Link Frequency/Voltage
			Link Width
			Hot Add
			D-states
	Summary
Chapter 4: Platform Power Management
	Platform Overview
		Common Platform Components
		Integration
			CPU Integration
			Chipset Integration
			Microservers and Server SoCs
		Platform Manageability
	CPU Sockets
		Node Controllers
	Memory Risers and Memory Buffer Chips
	Server Chipsets
		PCH and Platform Power Management
		PCH Power Management
		PCIe in Chipsets
		PCH Thermal Management
	Networking
		Ambient Temperature, TDP, and Thermal Management
		Attached Media
		LAN Power Management Features
			Media Speed
			Energy Efficient Ethernet
			Wake on LAN
			Active State Power Management (ASPM)
			Interrupt Moderation
	USB
		Link Power States
		Link Frequency/Voltage
	Storage
		Storage Servers and Power Management
		HDDs and SDDs
		SATA and SAS Drive Power Management
		Frequency/Voltage
		NVMe Drive Power Management
	Power Delivery
		Overview of Power Delivery
		Power Converter Basics
			System AC/DC Power Supply
				PSUs and the Boost Stage
				PSUs and the Isolated Buck Stage
				Redundant Power Supplies
				Shared Power Supplies
				PMBus
			DC to DC Power Converters
				Single-Phase Buck Converters
				Motherboard Multiphase Buck Converters
				SVID
				Motherboard Linear Regulators
				Integrated Voltage Regulators
				Power Management Integrated Circuit
		Power Conversion Losses
			Motherboard VRs
				Single-Phase Buck Converter
				Multiphase VR Losses
				Phase Shedding
				Diode Emulation and Burst Mode
			System Power Supplies (AC/DC)
				Right-Sizing Power Supplies
				Closed Loop System Throttling ( CLST)
				Losses in Redundant Power Supplies
				Power Supply Cold Redundancy
	Thermal Management
		System Considerations
		Component Thermal Management Features
			Processors
			Memory
		Platform Thermal Management
			Thermal Control Inputs—Sensors
				Voltage Regulators
				Power Supplies
		Fan Speed Control and Design
	Summary
Chapter 5: BIOS and Management Firmware
	BIOS Firmware
		Microcode Update
		Advanced Configuration and Power Interface
			S-states
			C-states
			P-states
			D-states
			ACPI Interfaces
		Setup Utility
	Management Firmware
		Node Manager Capabilities
			Hardware Protection
			Monitoring
			Power Capping
			Node Manager Policies
		IPMI
			Sensor Model
			System Event Log
			Node Manager API
		ACPI Power Metering Objects
	Summary
Chapter 6: Operating Systems
	Operating Systems
		C-state Control
			MWAIT
			HLT
		C-state Policy
			Processor Utilization
		P-state Control
			Software Controlled Interface
			Collaborative Interface
			Firmware Control
		P-state Policy
			Performance Capacity
			P-state Coordination
		T-state Control
		Global Power Policy
		Process Scheduling
			Topology and Capability Awareness
			Timer Tick Frequency
			Execution Consolidation (Core Parking)
				Energy Efficiency
				Power Capping
				Single-Threaded Performance
		Memory Management
		Device Drivers
			PCIe, SATA, and USB
			Graphics
	Virtualization
		Power State Control
			Idle Considerations
			Active Considerations
		Consolidation
		VM Migration
	Comparison of Operating Environments
		Microsoft Windows Server (including Hyper-V)
		Linux Distributions (including KVM)
		VMWare ESX and ESXi
	Summary
Chapter 7: Monitoring
	Hardware Monitoring
		Fixed Counters
		Core Performance Monitors
		Uncore Performance Monitors
			Global Freeze/Unfreeze
			Edge Detection and Average Time in State
			Standard Events and Occupancy Events
		Status Snapshots
		Counter Access and Counter Constraints
		Events and Metrics
			Time (RDTSC)
			Basic Performance
			Energy Use
			Temperature
			Frequency and Voltage
			C-States
			Memory Power and Performance
			PCIe Power Management
			QPI Power Management and Performance
	Management Controller Monitoring
		Component Power Sensors
		Synthetic Sensors
		Sensors and Events
	Software Monitoring
		Utilization and Processor Time
			Simultaneous Multithreading (SMT)
			Virtualization
		Processor Power State Requests
		Scheduler, Processes, and Threads
		Interrupts
		Memory
		I/O
	Tools
		Health Checks
			Turbostat (Linux)
			PowerTOP (Linux)
			Powercfg (Windows)
		Hardware Monitoring Tools
			Intel Performance Counter Monitor (PCM)
			Linux Perf
			IPMItool
		Operating System Monitoring Tools
			SAR
			Perfmon and Logman
	Summary
Chapter 8: Characterization and Optimization
	Workloads
		Identifying Suitable Workloads
		Workload Types
			Testing Tools
			Energy Efficiency Workloads
				SPECpower
				The Server Efficiency Rating Tool (SERT)
			Industry Workloads
			Idle Workloads
	System Characterization
		Steady State vs. Non-Steady State
		Data Collection
			Collection Duration
			Collection Frequency
			Event Ordering and Event Groups
			Multiple Tools
		Methodology
		Analysis
			Power Metrics
			Performance Metrics
	Optimization
		CPU Power Management
			P-States and Turbo
				Frequency Control in Windows
				Frequency Control in Linux
				Turbo Ratio Limit
				Uncore Frequency Scaling
			Core C-States
			Runtime Core Disable
			Package C-States
			Energy Performance Bias
			Hyperthreading
			Prefetchers
			PCIe
			QPI
		Memory
			CKE
			Self-Refresh
			Patrol Scrub
		NIC
		Storage
		Thermal Management
		Optimization at a Glance
	Summary
Chapter 9: Data Center Management
	Data Center Management and Power Distribution
	Data Center Facilities
		Power Infrastructure
			Power Distribution Efficiency
			Power Conditioning
			Back-up Systems
		Cooling Infrastructure
	Simplified Total Cost Models of Cost and Compute Infrastructure
	Performance per Watt per Dollar
	Summary
Appendix A: Technology and Terms
Index