PCI Express 6.0: High Bandwidth Interconnect

VIP Expert

Jan 12, 2022 / 2 min read

PCI-SIG® recently released the latest revision of the PCI Express® specification PCIe® 6.0. With 64GT/s raw data rate physical layer enabling up to 256 GB/s data transfers via 16-lane configuration. With this announcement PCIe continues to meet the industry’s need for high-bandwidth and low latency interconnect, whose potential could be leveraged by dependent storage (NVMe), and coherency (CXL) protocols.

PCIe 6.0 doubles the bandwidth from PCIe 5.0, which is achieved by moving away from differential signaling and using well established PAM4 modulation and gray coding techniques. Additionally, Smart Forward Error Correction and replay techniques provision low latency transfers between link partners.
The specification continues to maintain backward compatibility with existing device discovery and software ecosystem incorporating a lot of refinements in each layer compared to the previous generation.

A large number of the changes lie in data link layer including Flit sequence numbers, credit Flow control, and Flit structural definitions.

A new 256B Flit Mode, added with efficient packing, is allocated for TLPs, DLLP, CRC and ECC. The snapshot below captures Flit exchange transactions in the Synopsys Verification IP transaction log. As shown below, Flits with NOP, or Payload with or without DLLPs, are exchanged.

PCIe 6.0 Flit exchange transactions

The link layer changes introduced are directed at fixing issues with sequence number rollovers and shared/dedicated credits. To address sequence number rollovers, sequence numbers are incremented for Flits with payloads; all other Flits use the same sequence number as the previous Flit. To help facilitate this, Flit usage is updated to indicate Flits with and without payloads. The shared credit mechanism is tuned to prevent 1 VC from dominating all the shared credits. In Flit Mode, shared credits are the “default” with dedicated credits only used when required. The distinction between TLPs using shared/dedicated credits is managed using a Flit Mode prefix.

The physical layer changes bring in a certain level of backward incompatibilities due to gray coding and precoding rule changes. The LTSSM rules for performing loopback equalization at 64GT/s and entering compliance at 64GT/s are updated. Other LTSSM changes include new state transitions added to the recovery states in Flit Mode and tweaks to the 64GT/s equalization procedure.

Below is a snapshot of a VIP waveform illustrating the rate change to PCIe 6.0 with full equalization enabled between link partners.

PCIe 6.0 VIP waveform

A new power saving state L0p is applicable only in Flit Mode. L0p is part of conventional L0 state of LTSSM and uses configured link width for power savings. The snapshot below depicts L0p transition with Link width configured to 1 lane. Data transfer continues to take place even at this state.

PCIe 6.0 Flit Mode diagram

Updates in critical layers of the PCIe stack also brings several challenges with the functional verification of PCIe 6.0 designs scaling from IP to SoCs to systems.

Synopsys has been one of the key contributors to the PCIe specification and a board member in the PCI-SIG organization. Synopsys provides a complete PCIe 6.0 DesignWare IP solution and the industry’s first PCIe 6.0 VC Verification IP and Test Suite, almost 2 years ago to early adopters. Synopsys is a leading provider of a holistic system design and validation solutions using ZeBu transactors, Speed Adaptors, Hybrid and Virtual solutions.

Synopsys continues to provide the industry’s first and most comprehensive protocol continuum solutions.

Continue Reading