EDA
System
IP
SolvNetPlus
Training & Education
The rise of artificial intelligence is fundamentally altering server design.
With data center capacity increasingly dedicated to complex AI models, the industry must find ways to support these insatiable workloads. In kilowatts alone, the increase in power density is enormous: traditional data center racks typically draw 5 to 15 kW, but AI racks already exceed 100 kW. And some analyses project peak densities reaching 1,000 kW by 2029.
But the challenges extend beyond scale.
AI model training and inference workloads are forcing the industry to rethink not only how much compute fits in a rack, but how servers are architected from end to end — transforming computing infrastructure as we know it.
AI Doesn’t Scale by Accident
Explore the IP that enables high-performance, scalable AI systems
Why infrastructure needs an overhaul
Traditional servers can’t keep pace with the unique characteristics of AI. In a classical CPU-centric model, servers are optimized for general-purpose applications — web, database, email, ERP — and relatively modest data streams. A small number of powerful processors handle most computational tasks sequentially, stepping through instructions as each request arrives.
Today’s AI models behave very differently. They rely on dense calculations (matrix multiplication) distributed across thousands of compute cores running in parallel — the kind of work that GPUs, custom AI chips, and other accelerators are specifically designed to execute.
The scale of these tasks is staggering.
AI models with tens of billions — or hundreds of billions — of parameters can consume hundreds of gigabytes or even terabytes of memory. This requires continuous, high-volume data movement across layers of compute, memory, and storage. If those data paths are not carefully designed, expensive accelerators sit idle waiting for data and system efficiency drops as a result.
To address this growing memory bottleneck, server architectures are increasingly turning to disaggregated and pooled memory models enabled by technologies such as CXL 3.0. By allowing memory resources to be shared dynamically across multiple nodes, pooled memory helps overcome the “memory wall” and supports the massive context windows required by modern large language models — improving utilization while reducing the need to overprovision local memory.
Not all AI tasks are the same, either. Server architecture must support two distinct phases:
- Training: Compute-intensive processes that establish and refine AI model parameters, often running for weeks at a time on tightly coupled clusters.
- Inference: Always-on workloads that apply trained models to real data — answering questions, detecting anomalies, generating text and images, and performing other tasks — where low latency, predictable performance, and cost per request matter most.
While server architectures for training affect the speed at which models are improved, inference directly affects user experience and operating costs.
AI infrastructure must excel at both, which is precisely why an overhaul is needed. Adding more CPU-centric servers to the data center does not solve the problem. The architecture itself must be rethought to align with the realities of AI workloads, which require massive parallelism, extreme memory performance, and continuous data movement.
Re-centering servers around accelerators
What’s needed is a specialized stack designed around GPUs or other dedicated AI accelerators. In this new model, the accelerators do most of the heavy lifting, while CPUs assume more of a supporting role, handling task scheduling and resource allocation.
Leaning on accelerators doesn’t fully solve the problem, of course, because computing power is only as effective as the data paths that support it. An optimized AI server layout must therefore include:
- Compute nodes: Multiple accelerators tightly connected in a high-bandwidth mesh to facilitate rapid, synchronized communication.
- Shared memory pools: Ultra-high bandwidth local memory per accelerator, paired with low-latency shared memory across compute resources to minimize data movement and keep accelerators fed.
- High-speed interconnects: High-capacity, low-latency chip-to-chip and card-to-card links that provide the bandwidth required for distributed AI workloads.
By judiciously coupling compute with optimized data paths, designers can ensure efficient data flow from storage to memory to compute, minimizing latency and maximizing throughput for faster training cycles and better GPU utilization.
Delivering these architectures, however, requires seamless integration across processor, memory, and accelerator subsystems with tightly coupled interconnects — areas where system complexity and design challenges are growing rapidly.
Tailoring architecture for cloud, edge, and on-prem
While much of the industry is converging on acceleration-centric architectures, the ideal AI server is still not a one-size-fits-all solution. Specific types of deployment come with their own requirements and challenges.
- Cloud AI: These architectures prioritize the ability to instantly add or remove compute capability as demand fluctuates. They also rely on multi-tenancy and security isolation — technical safeguards that allow multiple different companies to run workloads on the same physical hardware without risking data leaks.
- On-premises: For organizations with consistent, large-scale workloads, keeping servers within their own data centers offers lower-latency control. This allows engineering teams to fine-tune the hardware for custom AI software and proprietary data unique to that business rather than relying on generalized cloud configurations.
- Edge AI: In factories, autonomous vehicles, and other edge environments, computing resources and power are often severely limited. Architecture must therefore focus on low-power accelerators and specialized data methodologies that store the most important information as close to the processors as possible, helping ensure ultra-fast responses and results.
Without these adapted architectures, companies risk overspending on cloud, underutilizing on‑prem assets, or missing low‑latency opportunities at the edge.
Managing thermal output and heterogeneous integration
Regardless of deployment model, all AI systems face growing physical and integration constraints.
As more compute power is packed into smaller systems, traditional fans struggle to dissipate the heat these systems generate. To prevent thermal throttling — a protective state where chips slow down and sacrifice performance to avoid heat damage — advanced cooling techniques are required. Liquid or immersion cooling is becoming a necessity, using cold plates on CPUs/GPUs or submerging hardware in non-conductive fluid to prevent thermal overload.
Another challenge is integrating CPUs, GPUs, and custom ASICs into a single, coherent system. While these heterogeneous chips excel at different tasks, extracting their full value requires more than fast hardware links. Designers must align the software, firmware, and runtime layers that manage scheduling, memory movement, and synchronization across devices.
Without careful tuning of AI frameworks, device drivers, and the connective software stack, data can bottleneck at handoff points, workloads may be poorly balanced, and accelerators can spend valuable cycles idle — ultimately degrading system-level performance.
A holistic, system-level approach
Building AI-ready servers is far more complex than simply assembling faster processors or adding more memory. It requires end-to-end architectural optimization — aligning compute, memory, interconnect, packaging, power delivery, thermal management, and the software stack that binds them together. As AI workloads continue to scale in size, concurrency, and power density, fragmented or siloed design approaches will increasingly fall short.
This is where Synopsys plays a critical role.
With deep expertise across silicon IP, advanced packaging, high bandwidth memory interfaces, interconnect technologies, and industry-leading, AI-powered electronic design automation (EDA), we help system designers move beyond incremental improvements toward truly cohesive AI server architectures. Our solutions enable teams to explore architectural tradeoffs early, integrate heterogeneous components efficiently, and validate complex systems before they reach silicon — reducing risk, accelerating time to market, and improving overall performance and energy efficiency.
Ultimately, the future of AI infrastructure depends on treating servers as unified, optimized systems rather than collections of discrete components.
Continue Reading
New Synopsys.ai Copilots Deliver 2–5× Faster Chip Design Productivity
