Synopsys and Ansys power the future of innovation—connecting silicon to systems.
EDA
System
IP
SolvNetPlus
Training & Education
Part 1: The Rack Becomes the New Unit of Compute
As the technology industry transitions from general-purpose compute to AI workload optimization, the design focus is shifting from the server to the rack.
The rack — not the servers within it — is what ultimately determines computing efficiency, now measured in tokens per watt or tokens per dollar. Improving that efficiency requires everything to be designed, integrated, and optimized at the rack level, from compute and memory to interconnects and power.
Bringing these elements closer together helps in obvious ways. Shorter interconnects and lower latency improve performance, while better power efficiency and space utilization lower total cost of ownership.
But those gains come with consequences too, and you cannot address them in isolation.
AI Doesn’t Scale by Accident
Explore the IP that enables high-performance, scalable AI systems
More density, more heat
Heat is one of the most significant consequences of higher performance racks, and also one of the hardest to address.
Adding more compute to an enclosure draws additional power into a fixed space, and every watt becomes heat you have to remove. That makes thermal management a whole-rack problem, not something that can be solved at the processor or server level.
Power consumption has already passed 100 kilowatts per rack, and it will likely reach a megawatt before the end of the decade. Air cooling remains viable with good airflow management up to roughly 50 kilowatts per rack; beyond that, it stops being practical. Liquid cooling is therefore becoming a baseline requirement. From cold plates to the plumbing and infrastructure around them, that cooling adds cost.
Moving to 800-volt distribution
Getting the heat out is only half the problem. You also have to get the power in, and the low-voltage approach that has always powered racks is reaching its limits.
Racks distribute power at a low, fixed voltage — around 54 volts — and at a fixed voltage, the only way to deliver more power is to push more current. Current is what dictates how much copper you need to carry it, and the losses rise with it. At 54 volts, a single megawatt-scale rack would demand up to 200 kilograms of copper busbar, crowding out space meant for compute while increasing power lost to resistive heating.
To feed next-generation racks, the industry is moving toward 800-volt distribution. Higher voltage carries the same power at a fraction of the current, so you need far less copper and the losses drop with it.
Shifting to 800-volt distribution is not a simple swap. The conversion stages, the safety model, and the thermal profile all change. And every interface on the rack must be designed and built for a hotter, higher-voltage environment.
Designing the rack as one system
The rack used to be little more than an enclosure for servers. Each server could be designed and optimized largely on its own, then installed alongside other servers that operated much the same way.
In an AI factory, the rack has a much more active role. Compute, memory, interconnects, power, cooling, and software must work together as a single, optimized system.
The workload dictates what that system must deliver. Compute has to provide the required processing capacity. Memory has to supply the capacity and bandwidth needed to keep the processors working. Interconnects have to move data among processors, memory, and other components with sufficient bandwidth, low latency, and power efficiency.
Meeting those requirements creates a set of closely connected design decisions. The amount and arrangement of processors affect power density and heat. Power delivery and cooling affect how much compute the rack can support. The placement of components affects the distance, speed, and energy required to move data. A change in one area can alter the conditions in another.
These elements can no longer be designed separately and assembled late in the process. The rack has to be architected from the top down, with the intended workloads providing the starting point. Training, inference, and agentic AI workflows place different demands on the system, and those demands must inform its electrical, thermal, mechanical, and computational design from the outset.
Rack-scale design begins at the silicon level
Designing the rack as one system requires the same integrated approach at every level within it, including the silicon. This is especially true as the semiconductor industry shifts from monolithic chips to multi-die designs.
A 2.5D or 3D package brings compute, memory, and I/O dies together so they can operate as a single device. In many ways, it resembles a miniature version of the rack. The dies must exchange data, receive power, and dissipate heat within much tighter dimensions. Their placement affects interconnect length, bandwidth, power consumption, heat flow,
and mechanical stress.
With 3D stacking, these relationships become more difficult to manage. Power and data have to move across multiple layers while heat is removed from dies buried within the stack. Cooling choices can affect mechanical stress, while power-delivery decisions can influence thermal behavior and the area available for signaling.
These chip-level decisions carry through to the rack. They affect how much power the system consumes, how much heat must be removed, and how efficiently its compute, memory, and interconnects work together. Designing the rack as one system therefore begins with the silicon and packaging inside it.
Simulation connects design decisions
Electrical, thermal, mechanical, and computational behavior now have to be evaluated together, from the multi-die package to the rack. Simulation allows engineers to see how a change at one level affects the others while the architecture can still be changed.
That is increasingly important as power density rises. Silicon behavior shapes rack-level power and cooling requirements, while rack constraints influence decisions inside chips and packages.
An AI factory cannot be assembled from independently optimized parts. Its performance depends on how well those parts operate as one system.
Part 1: The Rack Becomes the New Unit of Compute
Part 3: Compute Only Scales When Data Can Move (coming soon)
Part 4: What the Next AI Stack Makes Possible (coming soon)
Continue Reading
Architecting AI Factories: Rack-Scale Design Series
The Rise of AI Factories: Powering an Era of Pervasive Intelligence
