EDA
System
IP
SolvNetPlus
Training & Education
Synopsys IP
Technical Bulletin
In-depth technical articles, white papers, videos, webinars, product announcements and more.
AI infrastructure is undergoing a fundamental shift. While compute density per piece of silicon continues to rise via next-generation process geometries, advanced AI processors, and innovative cache memory technologies, AI system hardware performance is now constrained by maximum reticle‑size dies and by how architectures are evolving off the compute die. DRAM memory bandwidth, die‑to‑die connectivity, and scale‑up and scale‑out fabrics increasingly determine whether compute gains translate into usable performance.
For companies building large‑scale AI platforms, this marks a transition from compute‑centric optimization to IP‑centric system design. Interface IP has become a first‑order design consideration, governing bandwidth density, energy efficiency, latency, and ultimately scalability from package to rack to cluster.
This technical bulletin examines how AI workloads stress infrastructure, why balanced compute, memory and interconnect architectures are now mandatory, and how scalable, silicon-proven IP enables die-to-die and chip-to-chip interfaces for the next generation of AI systems.
Training and Inference Stress Infrastructure Differently—but Share the Same IP Constraint
Training and inference represent fundamentally different system optimization problems, yet both are converging on the same limitation: I/O efficiency.
Large‑scale training prioritizes synchronous scaling across thousands of XPUs. Collective operations, Mixture‑of‑Experts (MoE) routing, and heavy east‑west traffic place sustained pressure on scale‑up fabrics. While raw compute continues to grow, network bandwidth and link efficiency increasingly caps achievable scaling. HBM bandwidth remains critical, but fabric latency, bandwidth density, memory coherency and energy per bit often become dominant design constraints.
Inference shifts the focus toward latency per token and tokens per second per watt metrics. Long‑context models expand Keys and Values (KV) cache requirements, driving memory bandwidth and capacity to the forefront as these Keys and Values matrices may not be stored locally or may need to be recalculated across the hardware infrastructure. As inference deployments become distributed, network pressure returns—this time with highly variable, bursty traffic patterns that demand flexible, efficient interfaces.
Across both training and inference workloads, compute per XPU is increasing dramatically and scaled in a distributed system. This fundamentally creates problems and becomes a Theory of Constraints (TOC) problem where bottlenecks must be managed. Without proportional scaling in HBM, SerDes, and die‑to‑die connectivity, performance gains stall. Closing this compute‑to‑I/O gap is fundamentally an interface IP problem.
AI Doesn’t Scale by Accident
Explore the IP that enables high-performance, scalable AI systems
Training Workloads Are No Longer Reliably Compute Bound
Modern AI training spans a wide range of behaviors. Compute-bound, memory-bound, and network-bound constraints need to be understood to optimize today’s workload bottlenecks. Post‑training workflows such as reinforcement learning, preference optimization, and fine‑tuning introduce highly variable stress profiles. Pre‑training has expanded beyond dense models to include long‑context training, multimodal inputs, Mixture‑of‑Experts architectures, and simulation‑grounded world models.
What was once predictably compute‑bound is now frequently memory‑bound or network‑bound. The defining characteristic is variability—not just scale. As models become more sophisticated, infrastructure must support rapidly shifting demands across compute, memory, and network IP? subsystems.
The implication for system architects is clear: balanced architectures are mandatory. Compute, memory, and network bandwidth must scale together, driven not only by model size, but by increasing model complexity and diversity.
Table 1. Diversity of AI training workloads and their shifting compute, memory, and network bounds.
Inference Workloads Introduce Stage Specific Bottlenecks
Inference performance is shaped by multiple pipeline stages—prefill, decoding, routing, speculative execution, and reasoning—each stressing different parts of the system hardware.
Long‑context decoding and chain‑of‑thought prompting amplify memory bandwidth demand. Mixture‑of‑Experts routing and distributed serving increase sustained network load. Parallel sampling and speculative decoding add further variability. No single IP subsystem dominates across all inference scenarios.
For designers, this means infrastructure must support multiple operating routines efficiently. IP must deliver consistent bandwidth density, low energy per bit, and predictable latency across highly dynamic workloads—without over‑optimizing for any single use case.
Table 2. Mapping inference stages to compute, memory, and network bottlenecks and their projected scaling
Subscribe to the Synopsys IP Technical Bulletin
Includes in-depth technical articles, white papers, videos, upcoming webinars, product announcements and more.
Rack Level Density Is Driving Architectural Change
AI infrastructure for training and inference workloads is entering an era of extreme rack‑level density. Power budgets are rising from tens of kilowatts to hundreds of kilowatts—and toward megawatt‑class racks by the end of the decade.
This trend is driven by increasing numbers of reticle‑limited XPU dies per rack, aggressive co‑packaging, and the adoption of direct liquid cooling. As density increases, traditional 2D integration approaches become insufficient, accelerating the transition to multi-die designs and network scaling in multiple dimensions including front end networks, backend networks, scale up and scale out networks. These dynamic architectural decisions are changing at a rapid pace and have performance implications
Memory systems must evolve in parallel. As the ratio of compute silicon to HBM stacks increases, HBM bandwidth per stack must rise to preserve architectural balance. Interface IP must support higher pin speeds, denser signaling, and tighter power envelopes to sustain performance at the rack and cluster level.
Table 3: HPC Infrastructure Level Roadmap
Standards, Pseudo Standards, and Parallel Interface Evolution
Interface ecosystems are evolving along multiple parallel paths. In scale‑up fabrics, standards‑based such as UALink, ESUN and SUE, and proprietary approaches are advancing simultaneously to optimize per-wire bandwidth while providing critical memory awareness concepts easing compute expansion. Die‑to‑die connectivity continues to develop along both open and custom‑optimized lines to exceed reticle limitations and essentially utilize many heterogeneous and homogeneous dies as a single compute engine while expanding the number of lanes physically possible on a single die for added system bandwidth. Memory interfaces are similarly diversifying with the emergence of customized HBM variants alongside JEDEC standards to enable parallel links for KV cache bandwidth needs. Increasing bandwidths for the leading standards PCIe & CXL links off the host CPUs and associated infrastructure are driving next generation standards from 3 year to 2 year development cycles.
For system architects, product availability—not just standards ratification—now drives architectural decisions. Successful platforms must accommodate both standards alignment and early adoption of advanced capabilities. This places a premium on flexible, forward‑looking interface IP that can evolve alongside system requirements that are more often requiring customizations to meet the needs for AI systems with an edge in performance.
Figure 1. Evolution of standards and pseudo-standards
The Compute Density Playbook: Keeping I/O and Memory in Lockstep
Sustained AI performance depends on synchronizing compute, memory, and I/O scaling. As compute area per die increases and multiple reticle‑limited dies are co‑packaged, I/O and memory interfaces can consume a significant portion of total silicon area.
Higher‑speed SerDes, efficient custom HBM interfaces, and 3D integration help restore balance. By increasing bandwidth per pin and per watt, fewer memory stacks can support larger compute footprints. Multi-die design further improves energy efficiency per FLOP by shortening interconnect distances and enabling denser packaging.
This balancing translates into higher usable compute density—if interface IP is designed to scale coherently at the package level.
Figure 2. Compute Density Scaling from HBM3E to HBM5
Toward the Megawatt Class AI Rack
By the end of the decade, AI systems will support unprecedented compute density at the rack level. Megawatt‑class racks become feasible through the convergence of multi-die design, next‑generation HBM, and high‑bandwidth scale‑up and scale‑out fabrics.
Achieving this vision requires IP that delivers extreme bandwidth density at acceptable power levels, while maintaining low latency and adequate reliability. High‑speed scale‑up links, high-bandwidth die‑to‑die connectivity, and custom memory interfaces must operate as a tightly integrated system—validated across power, thermal, and signal integrity dimensions.
Figure 3: The 1MW Rack by 2029
IP Is the Foundation for Scalable AI Platforms
As AI infrastructure scales toward megawatt‑class racks, performance bottlenecks are shifting decisively off‑die. The ability to scale compute now depends on the efficiency, bandwidth density, reliability, and integration readiness of the IP that connects die to more die, compute to more compute, compute to memory, memory to more memory and everything to the scaled layers of networking of the cluster.
Synopsys addresses this challenge with a comprehensive portfolio of silicon-proven IP for HPC and AI designs, combined with industry‑leading AI-powered EDA tools and gold-standard multiphysics simulation and analysis. Together, we enable customers to design, verify, and deploy complex AI systems with confidence—optimizing signal integrity, power integrity, and thermal behavior as part of a unified system‑level approach.
By unifying scalable IP with end‑to‑end design and validation capabilities, Synopsys helps customers to close the compute‑to‑I/O gap and turn extreme compute density into deployable, efficient, and scalable AI platforms.
Continue Reading
224G PHY IP for AI Networks: Extending DAC Reach and Reliability for Next-Gen Data Centers