Synopsys and Ansys power the future of innovation—connecting silicon to systems.
EDA
System
IP
SolvNetPlus
Training & Education
Part 1: The Rack Becomes the New Unit of Compute
Part 2: Designing AI Systems at Rack Scale
AI factories bring large numbers of CPUs, GPUs, and other AI accelerators together to train models, run inference, and execute other AI workloads at scale. Their combined processing capacity is enormous, but the system can only deliver that performance if data moves fast enough to keep pace.
These processors must exchange data with memory, with each other, and with components throughout the rack. If the interfaces cannot move that data at the required speed and efficiency, processors spend time waiting instead of working.
AI Doesn’t Scale by Accident
Explore the IP that enables high-performance, scalable AI systems
Bandwidth, latency, and energy per bit
Interface performance is shaped by three primary measures: bandwidth, latency, and energy per bit.
Bandwidth determines how much data a connection can carry. Latency determines how long the transfer takes. And energy per bit measures the power consumed to move that data.
Each contributes to the performance and economics of the rack.
An interface may provide high bandwidth but introduce too much latency for a particular workload. Another may deliver the required speed but use too much power to scale across thousands of links. Every watt consumed in data movement also adds to the rack’s overall power and thermal load.
As AI processing demands increase and AI factories scale, interfaces must move more data, use less power, and maintain signal integrity at higher speeds and over longer distances. Reliability must be maintained as well.
These are not independent targets. Improving one can make another more difficult to achieve.
The right balance depends on what the system is being designed to do. Training, inference, and other AI workload tasks place different demands on bandwidth, latency, and power efficiency.
The interfaces have to be selected and optimized around those demands.
Connecting every level of the system
Data moves across many types of connections in an AI system. Die-to-die interfaces connect components within multi-die designs. Memory interfaces connect processors to high-bandwidth memory (HBM) and other memory resources. High-speed SerDes and PCIe links connect chips and components throughout the server and rack. Rack-scale fabrics and rack-to-rack links extend those connections across the AI factory.
Each link operates over a different distance and serves a different purpose. A die-to-die interface within a package does not have the same requirements as a connection that crosses the rack. The differences affect the data rate, reach, latency, power, area, and signaling technology required at each level.
The shift to multi-die systems adds more connections inside the silicon itself. Compute cores can consume much of the main die, pushing memory and I/O onto separate dies. Data then has to move efficiently among compute dies, I/O dies, and HBM before it travels through the rest of the system.
Interconnects are therefore becoming more influential components of both the silicon and system architecture. They are no longer limited to communication between chips or servers. Their role begins inside the package and extends across the AI factory.
Extending connectivity with optics
Distance becomes more difficult to manage as bandwidth rises. Electrical signaling over copper requires more power and more complex signal management as connections become faster and longer.
That is increasing the importance of optical connectivity. The industry is moving from pluggable optical modules toward co-packaged optics, bringing optical connections closer to the compute and networking silicon.
Copper will continue to serve many shorter connections. Optics can carry more of the load as distance and bandwidth make electrical links less efficient. Where that transition occurs will depend on the architecture, workload, and economics of each system.
The choice becomes especially important as AI factories scale beyond a single rack. Rack-to-rack connections must move enough data for large groups of processors to work together without the network becoming the bottleneck.
Designing interface IP for the system
Die-to-die interfaces, memory interfaces, and high-speed SerDes provide the foundation for data movement. Many are built around industry standards that promote compatibility and broad adoption. AI systems are also creating requirements that extend beyond a standard configuration.
Some customers need to push performance for a specific workload or architecture. Others need to reduce silicon area or power by removing capabilities their designs do not require. The appropriate interface depends on the system’s performance, power, area, latency, reach, reliability, and cost targets.
That is driving greater customization of interface IP. The IP has to be optimized with the chip, subsystem, workload, and larger system requirements in mind.
At Synopsys, we work closely with customers to customize our interface IP around their requirements, integrate it into their chips and subsystems, and validate its operation under the intended workloads. That system-level view helps ensure the interface delivers the required capability within the chip, rack, or larger system.
The amount of compute inside AI factories will continue to grow. How much of it can be used depends on whether data reaches the right processor at the right time at an acceptable power cost. The interfaces determine whether processors remain productive or wait for the next transfer.
Compute only scales when the connections scale with it.
Part 1: The Rack Becomes the New Unit of Compute
Part 2: Designing AI Systems at Rack Scale
Part 4: What the Next AI Stack Makes Possible (coming soon)
Continue Reading
Architecting AI Factories: Rack-Scale Design Series
