Build Blog
The rise of Thermal Design Power
A CPU is a CPU is a CPU. My shallow dive into datacenter infrastructure. From transistors to 700W sockets.
Increased demand for compute has led to a wave of rising TDP, which is altering the course of datacenter design and, by extension, the cloud platforms we all run on.
A CPU or Central Processing Unit is a packaged silicon die or dies fabricated using the most advanced manufacturing methods of its time.
There are happy people, and then there are those who have found themselves wandering in the unending corridors of a modern datacenter. Inside, your ears fill not with the buzz of excitement, but the eternal drone of transistors and fans and flywheels, providing internet-connected compute over laser-lit fibers to all four of those pointy corners of our globe.
It is precisely in this calculable conversion of kVA (or kW), the only recurring bill a datacenter charges their customers, into goods or services that the responsibilities of a CPU lie.
You see, even with GPUs, CGRAs, FPGAs, DSPs, VPUs, RDUs, and on and on, the CPU still remains as the executor of processes with sequential dependencies.
In this post, I seek to provide some insight into the machinations of cloud computing and the winds that shape it.
Why Thermal Design Power
Let's talk transistors. A transistor is a foundational building block of digital logic. They are described in text files and etched into the silicon wafers in every foundry.
I like to picture a drawbridge with an input track, an output track, and a control signal. If the control signal isn't present, the drawbridge stays up, and nothing flows along the track. Applying the control signal lowers the bridge, and enables traffic to flow.
However, in this case, both the traffic and control signals are electrons. And our trusty control signal needs enough energy to operate the drawbridge. Not enough voltage, and the bridge won't lower.
There are also losses in the form of heat. This is of vital importance. Silicon transistors are not superconductors. When the bridge lowers to let the electrons flow, they slow down on their passage, heating up the bridge.
The management of the heat dissipation requirements of an entire package leads to a manufacturer published number, in watts, called the Thermal Design Power, or TDP.
Watts in, Watts out
Now hold on. Watts is a unit of measurement of power or energy transfer. Why would it be used to inform heat dissipation?
Well, probably all of the energy going into a CPU will become heat. And most of that heat creates a temperature gradient in the package. The more energy, the more heat, the higher gradient.
Your CPU can be deployed in a location that has air. With enough transfer of heat to the surrounding air, a package can be kept within its designed operating envelope. If no air is present, or does not suffice to dissipate enough energy in the form of heat, liquid can be circulated across the package or large solid conductive elements can be utilized.
One widely adopted practice is to place a copper or aluminum element with thermal interface material on the package and then use fans to pull or push air through the element.
You're History
I don't need a graph to show you that core counts are rising fast and TDP along with them.
The E5-2699A is a ten year old discontinued CPU package manufactured by Intel. It has 22 cores and 44 threads. It has a TDP of 145W.
The 7713 is a five year old CPU package manufactured by AMD. It has 64 cores and 128 threads. It has a TDP of 225W.
The 6980P is a two year old CPU package manufactured by Intel. It has 128 cores and 256 threads. It has a TDP of 500W.
The Zen 6 flagship is an unreleased CPU package manufactured by AMD. I believe it will have 256 cores and 512 threads. I believe it will have a TDP of 700W.
You get the picture. Of course if you have enough pull as a cloud provider, you can just ask for your own CPU. X8i or MI300C anyone?
A box of pizza
Pizza box server. This is a term currently used to refer to a 1U in height, 19" wide, arbitrary depth air-cooled steel chassis.
Inside lie one or more Power Supply Units (PSUs), drive bays, fans, and the motherboard itself. Built into the motherboard are one or more CPU sockets. The Power Supply Units are responsible for rectifying the supply voltage down to the logic voltage required by the CPU, motherboard, drives, and fans.
The fans have the vital role of ensuring adequate air passes through the chassis. Our once-popular 1U fans are being made obsolete by the ever increasing airflow demands of the new power-hungry sockets. In addition, the higher per-server power draw combined with power-per-rack limitations cause per-server height to be a retreating concern.
Thus the Pizza box server is being phased out, replaced by 2U systems with their bigger fan blades or liquid cooled monoliths with individual servers no longer discernible.
I want to take a step back and take a look at the datacenter as a whole.
Get that heat out of here!
Energy enters the servers in the rack as electricity, and leaves as hotter air. Simple enough, right? Except now that we've got this heat into the air, we've suddenly got to get the air out of the building.
How do we get the air out of the building? If you've got a cross wind, just open up all the walls. (That's what certain hyperscalers do.) If you can't open the walls, then you need industrial heat rejection infrastructure.
The efficiency of this whole process is modelled as Power Usage Effectiveness, or PUE. PUE is the ratio of the input power to the building, to the input power of the system running the workload. A datacenter on Everest would have a PUE of 1.0000, as the inexorable mountain of ice would take on the monumental task of keeping the air within the designed operating envelope.
Thanks to Goodhart's law, we have hyperscalers playing a game of one-upmanship with this metric each time they find a better way to abuse it. Efficiency is still very important, and heat reuse is vital to making datacenters at sea level cause slightly less you know what.
But a datacenter standing alone is quite useless, the internet is a network of services after all. Now that we have the power to serve, how are we going to run our workloads?
Functions vs Infrastructure vs Platforms
If you can represent your entire internet service as individual functions operating within a finite window of context with a bounded execution window, Functions as a Service (FaaS) is the product for you. Seriously though, aside from the uncontrollable usage-based pricing and extreme vendor lock-in, FaaS does provide the ability to not care about the infrastructure you are executing on.
Now let's say you want to go the other direction. IaaC is the new hotness and you've just brushed up on the latest and greatest by watching a four year old tutorial video. Let's go and wrangle some servers, or VMs or clusters or whatever it is hyperscalers are backing these days. Just make sure you check the right infrastructure definition files into version control and don't accidentally wipe a production database with a typo. Or your snapshots along with it. Or just get billed for people accessing your private datastores, that's impossible, right?
Platforms like Build provide a unified abstraction to the underlying infrastructure. We enable continous deployment of enterprise application architectures combined with a friendly developer experience. We adopt open standards instead of locking you in to hundreds of proprietary services. Our access controls are straightforward and readable and don't require navigating a maze of identity inheritance. The cli enables agents to deploy and conviently lacks the ability to destroy all of your backups. And our 24-hour dev team will hop on a video call to assist at your convenience. No wading through different support tiers.
All in a day's work
The subject of TDP has been our periscope through which to view the high seas of the burgeoning datacenter industry. We have been able to peel back each layer of the stack to reveal the physics which shape web application deployment.
CPU power is only going up, and with it the cooling, custom packaging, and the sheer complexity of the servers containing them. As the hardware becomes increasingly elaborate, it is vital that developers can rely on platforms that let them focus on building software.
Lex Siegel is the founder of Build.io, the full-stack cloud platform for product teams to deploy and run production applications without managing infrastructure.