Cloud native EDA tools & pre-optimized hardware platforms
Ethernet has been a foundational technology since the 1980s. In the early days, workstations and PCs connected to on-site servers with a 10Mbps shared LAN using coax cables. Since then, Ethernet has evolved to support twisted pair and fiber optic cabling. The rates have increased 100Mbps and 100Gbps, leading up to the emerging 1.6Tbps Ethernet standard.
As Ethernet speeds increase, so too have its applications diversified. From Voice over IP and video streaming to multiroom audio, industrial control networks, and even in-car networks.This progression has driven the need beyond simple packet transfer, to reliable packet transfer. Defining quality of service is critical for data streams that are particularly sensitive to loss and delay. This technical bulletin will take a dive into the need for 1.6T data transfers, the standardization efforts from IEEE’s 802.3dj group, an overview of the components of a 1.6T Ethernet Subsystem, and FEC considerations for the Ethernet controller required to handle all this data.
Figure 1: The gigantic world of Ethernet (Ethernet Alliance)
One application though is already pushing to these speeds – interprocessor communication. A single device has a finite limit to its processing capacity. Even with the latest processors or specialized machine learning accelerators, devices are still limited by the manufacturable chip size. However, when chips are combined, the sky is the limit! This finite limit to processing capacities underscores the need for a generation of Ethernet capable of Terabit speeds and minimal latency – marking the first application for 1.6T Ethernet. The first generation of applications is expected to be followed by intra datacenter switch-to-switch connections, enabling the pooling of high-performance processors and memory, boosting scalability and efficiency within cloud computing.
It’s essential that each node on the network adheres to the same rules defined by a set of standards to achieve effective communication. The IEEE has been responsible for the Ethernet standards since its inception. The 802.3dj group is currently formulating the latest iteration of the Ethernet standard, outlining the physical layers and management parameters for operations at 200G, 400G, 800G and 1.6 Terabits per second.
The objective of the 802.3dj group is to support an Ethernet MAC data rate of 1.6 Tbps with:
For the physical layer, the specification to support 1.6Tbps includes transmission:
The outlook is for the standard to be finalized by spring 2026. However, we anticipate the completion of the baseline set of features in 2024.
Let‘s dive into the components of a 1.6 Terabit Ethernet subsystem – specifically, the elements used to implement the Ethernet interface within the silicon of an ASIC or ASSP.
Figure 2: Diagram depicting the components of a 1.6T Ethernet Subsystem
At the top we have the networked applications which can be either on client machines or on compute or file servers. They are the source and the destination of all the Ethernet traffic. A special kind of application is the Ethernet Bridge or Layer 2 switch – this is neither the source nor destination of Ethernet traffic, but rather is an intermediate point along the way that forwards packets per rules defined in 802.1d.
Individual applications, or instances, connect to and from the Ethernet controller by one or more queues. The queue is most likely buffering traffic to or from the application, balancing the network‘s performance with that of the client or server. For maximum performance, the speed of the network should match the rate at which traffic is generated or consumed. This way we will minimize the delay as packets are exchanged end to end between the applications.
The Ethernet controller typically comprises a MAC and a PCS, although often this is simply referred to as the Ethernet MAC. Below the PCS we have the Attachment Unit Interface (AUI)– some readers might remember the D-type connector on the back of a workstation into which an AUI cable was plugged. That interface is still there in today‘s Ethernet, it‘s just faster. Finally, further down the stack, we can find the blocks responsible for controlling and managing the physical elements of the network, whether it is fiber optic, copper cabling or a backplane.
As shown on Figure 3, below the application and queues lies the Media Access Controller (MAC). The MAC manages Ethernet framing – looking after source and destination addresses, managing the length of the frame, adding padding, if necessary (in the case of a very short payload), and adding / checking the Frame Check Sequence (FCS) to ensure the integrity of the frame.
Figure 3: MAC frame format and Length: An Octet Breakdown
MAC variations can be categorized into 2 main types:
One type of MAC is found on a NIC which sits in a client, server or router. These MACs handle the essential task of terminating the Ethernet layer by adding and stripping the Ethernet specific overhead as the payload passes down and back up the stack. An integral function includes adding and checking the Frame Check Sequence (FCS) to safeguard data integrity. If any corruption is detected upon receive, the frame will be dropped. Additionally, the MAC in a NIC will be checking the destination address of the frame, ensuring accurate delivery within the network. The payload will most likely be an IP (Internet Protocol) packet.
NIC’s used to be the function of a plug-in card, hence their name: Network Interface Card. The card implemented the MAC, PCS and PHY– where queueing and any other intelligence was handled by the host processor. Today, we see smartNICs that can offload many networking functions, but they still maintain the same MAC layer.
On the other hand, switching or bridging MACs have a different approach. Here, the entire Ethernet frame is passed between the MAC and the upper layer. The MAC is responsible for adding and checking the FCS and statistics gathering for the likes of Remote Network Monitoring (RMON) support. Conceptually, an Ethernet Switch can be viewed as a dedicated application designed for this purpose. Despite being predominantly hardware-implemented to guarantee optimal wire rate performance, each of its ports incorporates a dedicated MAC. Although these ports may operate at different speeds, any rate adaptation is managed in the queues above the MAC layer.
Figure 4: Diagram showcasing the MAC, PCS and PMA connecting to the AUI
Figure 5: Diagram depicting components for Controller, PHY and Cables for a 1.6T Ethernet Subsystem
Figure 6: Diagram showcasing a MAC and PCS optical TX/RX separated by a length of fiber
The PCS has a bit error rate of 10^-5 on the link connected to the optical module plus additional errors introduced on the optical link. If we only implement a single RS-FEC end to end in this system, the resultant error rate would not meet the 10^-13 Ethernet requirement. The link would be classed as unreliable. An alternative would be implementing a separate RS FEC on each hop, the RS FEC will be encoded and decoded three times. Once at the transmit PCS, then in the optical module and again at the far link from the optical module to the remote PCS. This implementation would be very costly and would increase the end-to-end latency.
Integrating a concatenated Hamming Code FEC to the optical link is an optimal solution that meets the Ethernet requirements and is also well suited for random errors encountered on optical connections. This inner FEC layer creates an additional expansion of the line rate from 212 Gbps to 226 Gbps, thus it is essential that the SerDes can support this line rate.
Figure 7: Diagram depicting a full 1.6T Ethernet subsystem and the latency path
When focusing on minimizing latency in an Ethernet subsystem (specifically at the Ethernet interface level, not the overall network), it’s crucial to consider the specific circumstances - for instance, when both the packet source and the sink are running at a matched and high data rate. Conversely, in a trunk connection like those between switches, latency becomes less of a concern due to the more pronounced delays on slower client links. Similarly, when dealing with longer distances, the inherent delay due to the distance will dominate.
Furthermore, it’s worth noting that Time Sensitive Networking (TSN) addresses deterministic latency. In this context, a maximum upper latency for mission-critical applications is established, especially for lower speed or shared-infrastructure networks. Of course, this doesn’t mean that we should overlook latency in other scenarios. Minimizing latency remains a constant objective. Firstly, the cumulative end to end latency increases with each successive hop. Secondly, added latency often indicates additional circuitry or processing in the controller, which could lead to increased power consumption in the system.
To begin, we’ll set aside any queuing latency, and assume there’s a clear path from the application to the Ethernet controller without any contention for bandwidth. Bandwidth differences will result in packet queuing delays which should be avoided when latency is critical. Ethernet frames are built or modified on the fly as the packet pass through the transmit controller. Notably, significant storage is not required through line encoding and the transmit FEC phases.
Transmit message processing latency hinges on the particular implementation, but can be minimized with good design practices. The time taken to transmit a message is simply down to the Ethernet rate and the frame size. For 1.6T Ethernet, transmitting a minimum sized packet takes 0.4ns – essentially, one Ethernet frame per tick of a 2.5 GHz clock. The transmission time for a standard maximum size Ethernet frame is 8ns, extending to 48ns for Jumbo Frames.
When considering the time to traverse the medium, optical fiber latency stands at roughly 5ns per meter, while copper cabling is slightly faster at 4ns per meter. Although the message receive time is the same as the transmit time, it’s usually ignored since both processes occur simultaneously.
Even with the most optimized designs, latency due to the RS FEC decoder is unavoidable. To begin error correction, 4 codewords must be received and stored, which takes 12.8ns at a 1.6Tbps. Subsequent processes, such as executing the FEC algorithm, correcting errors (as necessary), buffering and clock domain management, further contribute to the controller’s receive latency. While FEC codeword storage time is a constant factor, the latency during message receipt is implementation specific, but can be optimized through good digital design practices.
In essence, there is an inherent, unavoidable, latency due to the FEC mechanism and the physical distance or cable length. Beyond these factors, good design practices play a pivotal role to minimize latency due to the Ethernet controller. Leveraging an integrated complete solution that includes the MAC, PCS and PHY, along with an expert design team, paves the way for the most efficient, low latency implementation.
Figure 8: First-pass Silicon Success for Synopsys 224G Ethernet PHY IP in 3nm process showcasing highly linear PAM-4 eyes
1.6 Tbps Ethernet caters to the most bandwidth intensive, latency sensitive applications. With the advent of 224G SerDes technology, together with MAC and PCS IP developments, complete, off-the-shelf solutions are available aligning with the evolving 1.6T Ethernet standards. Controller latency is all critical in 1.6Tbps applications. Beyond the inherent latency due to the protocol and error correction mechanism, the IP digital design must be meticulously engineered by expert design teams to prevent unnecessary latency being added to the datapath.
A silicon-proven solution requires an optimized architecture and precise digital design, emphasizing power efficiency and reducing the silicon footprint to make 1.6T data rates a reality. Synopsys silicon-proven 224G Ethernet PHY IP has set the stage for the 1.6T MAC/PCS possible. Using leading-edge design, analysis, simulation, and measurement techniques, Synopsys continues to deliver exceptional signal integrity and jitter performance, with a complete Ethernet solution including MAC+PCS+PHY.