Demystifying 40 Gigabit Ethernet Physical Layer Interfaces in Data Centers

Networking Chassis

Networking chassis have one or more data planes (a.k.a. line cards) with port side ASIC(s) that connect the chassis through port side (a.k.a. faceplate) modules to external networking devices on one end, and a fabric interface ASIC on the opposite end of the line card for connection to the fabric card(s) and other line cards through a backplane or a mid-plane interconnect.

Figure 2 shows the three primary 40 Gbps (10 GE aggregated) IC interfaces within a chassis. The chip-to-chip, chip-to-module, and chip-to-backplane interfaces have unique channel characteristics and electrical specifications in the IEEE 802.3 standard.

1. Chip-to-chip port side interface: XLAUI (40 Gbps Attachment Unit Interface) connects to the port side modules that have retimers on both transmitter and receiver sides
2. Chip-to-module direct attach interface: XPPI (40 Gbps Parallel Physical Interface) directly connects to the port side modules that do not include retimers on their electrical input or output interfaces
3. Chip-to-backplane interface: 40GBASE-KR4 (40 Gbps Backplane Interface) drives aggregated 40 Gbps (4x10 Gbps) signals across a one meter 10 Gbps capable backplane or mid-plane in a chassis

Chip-to-chip port side interface

A chip-to-chip interface is the simplest form of an interconnect that mainly consists of a short printed circuit board (PCB) trace with one or no connector. This interface can be either between two ICs on the same PCB plane or between a port ASIC and a module device with a signal conditioning IC that has data retiming functionality.

The first 10 Gigabit small form factor (SFF) pluggable module, XFP, was developed by the SFF Multi Source Agreement (MSA) group that defined a serial 10 Gbps electrical interface called XFI. XFI is used between the XFP form factor transceiver (transmitter and receiver functionality) module and a port side ASIC. The XFP module integrates the retiming function of both transmit and receive data paths at the expense of module size, power and cost. XFI is one of the simplest serial 10 GE chip-to-chip electrical interfaces between a port IC and an XFP module.

The XFI electrical interface supports PHY connections to both Multi-Mode Fiber (MMF)- and Single-Mode Fiber (SMF)-based XFP form factor modules. XFP optical modules are short reach 10GBASE-SR (MMF 2m to 400m), long reach 10GBASE-LR (SMF 10km), and extended long reach 10GBASE-ER (SMF 40km), all of which are defined in 802.3. Typically only short reach devices are used within a data center for inter-chassis connections.

The first IEEE 802.3 standard defined electrical interfaces to a 40 GE optical module was the XLAUI (40 Gbps Attachment Unit Interface) that is based on XFI electrical interface to an XFP transceiver. As a result, the original 40 GE modules, such as the MSA-defined C Form-factor Pluggable (CFP), were required to perform data retiming functions. A CFP module is quite large in size, would not support stacking or ganging, consumes high power, and does not allow high levels of port side density improvements beyond what is offered by 10 GE XFP modules. The same line card that supported 16 ports of a 10 GE module could only support up to 4 ports of 40 GE CFP modules. Even with the enhancements in CFP module design since its original development, CFP has not been broadly adopted in data centers, and IT managers have difficulty justifying upgrading their existing 10 GE solutions to these new 40 GE modules.

The XLAUI electrical interface supports PHY connections to both MMF and SMF CFP based optical modules, such as the IEEE 802.3 defined short reach 40GBASE-SR4 (MMF 100 to 150m) and long reach 40GBASE-LR4 (SMF 10km).

Chip-to-module direct attach interface

A chip-to-module interface consists of a short PCB trace and a module connector between a port side IC and a module that is without retiming capability.

To increase the faceplate densities to achieve the higher bandwidth that is required in chassis, the burden of the signal conditioning function that traditionally resided inside a 10 Gigabit pluggable module, such as XFI, was moved to the port side IC that directly interfaces the faceplate module. As a result, a new high-speed 10 GE serial electrical interface called SerDes Framer Interface (SFI) was defined by SFF MSA in SFF-8431 for an interface between a host ASIC and the enhanced small form-factor pluggable module, SFP+ (Figure 3).

The SFP+ module form factor is 30% smaller than its predecessor, the SFP optical transceiver. It uses less power, requires fewer components, supports module stacking, and is less expensive than XFP. SFP+ improves the port side densities and reduces the overall 10 GE system cost. However, it puts new burden on the PHY’s design in the interfacing IC to the module. Compared to XFI, SFI electrical has higher requirements on the transmitter side, the output eye is smaller, the incoming jitter from the SFP+ module is higher, and issues such as pulse width shrinkage due to pattern dependency would need to be addressed by the interfacing device that is referred to as the host device in the electrical specification that defines SFI and SFP+, SFF-8431.

SFI is defined for both limiting and linear mode modules. In the limiting mode, SFI supports PHY connections to the limiting SFP+ optical modules such as the short reach 10GBASE-SR (MMF 300m), the long reach 10GBASE-LR (SMF 10km), and the extended long reach 10GBASE-ER (SMF 40km), that have limiting receivers. In the linear mode version of the SFI, further signal conditioning capabilities are required by the interface PHY to compensate for electrical dispersion. The linear version of SFI supports connection to both the optical 10GBASE-LRM (MMF 220m) that has a linear receiver, and the passive Direct Attached Copper (DAC) TwinAx cabling (1m to 7m) that is a linear media.

To improve the port side densities in a chassis, the new XLPPI (40 Gbps Parallel Physical Interface) electrical specification was defined by IEEE 802.3 for a direct connection to a Quad Small Form Factor Pluggable (QSFP or QSFP+) module, as defined in SFF-8436. By using these smaller form factor modules in the port side, more 10 Gbps modules can be placed in the same space. This will allow more 10G connections to the face plate of a chassis. QSFP offers three times the density of the traditional SFP port and is lower in both power and cost than a 40 Gbps CFP solution. QSFP also supports module stacking to enable higher port side densities. Unlike CFP, QSFP supports copper cable connections in addition to optical interconnect that are attractive lower cost options for short-reach interconnects within a data center.

XLPPI is derived from the SFI electrical interface and therefore places higher signal integrity requirements on the interfacing PHY. However, few of the SFI parameters were further relaxed in this 10 Gigabit aggregated interface so that the same XLPPI electrical definition would cover both the optical and copper based modules. Therefore, XLPPI is the electrical specification to both passive copper based 40GBASE-CR4 QSFP+ module (TwinAx copper cable, 1 to 7m) and the optical modules such as the short reach 40GBASE-SR4 (MMF 150m) and the long reach 40GBASE-LR4 (SMF 10km).

As shown in Figure 5, QSFP module densities are able to address the growing port side bandwidth demands of a data center better than the original 40 GE CFP modules.

Since its initial definition, the CFP module has evolved over time and is now capable of integrating more than just a single 40 GE channel in its large form factor. It is now able to accommodate two or even three 40 GE ports for an equivalent throughput of 120 GE. However, unlike QSFP, CFP cannot be stacked and ganged due to its large heat sinks. However, CFP modules can be placed on the two opposite sides of a PCB on the port side edge to create an equivalent two row interface. Yet, even with all these CFP enhancements, only QSFP is capable of the highest 40 GE port side densities.

Chip-to-backplane interfaces

Demand for bandwidth increases does not only impact the port side module interfaces, but is an overall system issue that needs to be addressed. As shown in Figure 2, data packet enters a chassis through the port module, travels through the port side ASIC, processing ASIC, and backplane interface ASIC before it reaches the backplane interface. Then it crosses a backplane channel to a fabric ASIC and back across the same or a second line card to the output module for switching purposes. Beside the port side chip-to-chip or chip-to-module interfaces, each of the ASICs along the way needs to manage and process the increased bandwidth through each of their interfaces.

A chip-to-backplane channel consists of a much longer PCB trace and two backplane connectors between two ASICs that directly drive it from each end. A backplane channel is one of the most complex interfaces in a chassis, as each of its components impact the quality of the signal due to higher levels of trace densities, loss, and discontinuities. Therefore, the channel’s driving PHY needs to have enough transmitter and receiver equalization to compensate for these effects.

As bandwidth densities increase across the backplane and mid-plane, system designers are moving away from using aggregated XAUI and are upgrading to the serial 10GBASE-KR as defined by IEEE 802.3ap-2007 standard. The four sets of XAUI traces are replaced by a single set of 10GBASE-KR trace for 4X higher density. However, due to the faster data rates and the increase in channel densities, both the PCB materials used in the backplane and the line cards, along with the backplane connectors, may need to be upgraded to support 10 GE serial data rates across a one meter channel. PCB material properties such as dielectric constant and dielectric loss can have a major effect on the variation in channel loss. Legacy connectors can be a cause of major reflection and noise in the channel, impairing the quality of signal across the channel.

For the 40 GE backplane interface, IEEE 802.3ba defined the electrical specification for the 40GBASE-KR4 interface that is based on the aggregated 10GBASE-KR to support four lanes of 10GBASE-KR across a backplane to further increase the channel densities.

Conclusion

System and IC designers must understand the differences between the two port sides and the backplane electrical interfaces to select the SerDes PHY with the right capabilities for their respective applications.

On the port side of a chassis, XFI and SFI are the two major single lane 10 GE electrical specifications defined for the XFP and SFP+ module form factors. On aggregated 10 GE, XLAUI and XLPPI electrical specifications are defined for the 40 GE CFP and QSFP module form factors.

Even though a PHY capable of driving a backplane may seem to be a superset PHY capable of driving both the port side and backplane channels, if the PHY is not designed and verified to support all three interfaces, it will not fully meet the electrical specifications required to drive a port side module and a backplane channel under all conditions and its use will result in interoperability issues, as each of these specifications have subtle requirement differences that must be designed to and verified.

IC designers need reliable interface IP that is capable of supporting all three different electrical interfaces in a single SerDes PHY, independent of the channel type that is verified and licensed from a single IP vendor. This combination provides the ultimate flexibility, optimal cost and ROI, and time to market. Synopsys’ silicon proven DesignWare Enterprise PHY IP has the necessary features and capabilities to drive all three different 40 GE interfaces necessary for ASICs applications in data centers.