The machines are running, and almost no one has measured how badly. The waste is invisible because it is distributed: a few points here, a few points there, eight thousand exhausted specialists each losing a Monday.
It is 2:17 in the morning in Osaka, and Priya is on her 847th configuration. The chip on her bench is a dataflow accelerator from a Taiwanese fabless startup, twelve weeks out of the fab, in no public benchmark and no compiler toolchain that knows it exists. Her job is to make it run her customer's inference workload by Monday. She types a number into a configuration file and waits for the workload to compile and run. The monitor reads thirty-four percent of the chip's rated throughput. She changes a tiling parameter. Forty-one. She changes the memory layout. Thirty-eight. The chip is capable of eighty percent on this workload and she knows it, and the gap is not a mystery to her; it is the cost of a part whose behavior no one has yet encoded into the tools she is using, and that encoding is weeks of work she does not have. At 2:31 she commits the number that gave her forty-one and goes to bed. The chip will run at forty-one until someone has the time to do it properly, which is to say, in the experience of everyone in this profession, never.
Priya's profession does not have a clean name. The people in it are called performance engineers, or kernel engineers, or compute architects, or, in the job postings that cannot quite describe what they do, heterogeneous systems specialists. What they share is a rare and slow-built skill: the ability to take a piece of silicon whose behavior is documented badly or not at all and coax a workload onto it at something near its potential. There are, on the most generous estimate, about eight thousand of them on Earth who can do this for a genuinely new architecture, and that number is the binding constraint on everything that follows.4Binding-constraint specialist pool ~8,000 (2025); this is the count who can integrate a genuinely new architecture by hand, not the broader pool of kernel engineers for dominant-vendor chips. [internal estimate] It is not the count of people who can write a kernel for the dominant vendor's chips, which is large and growing. It is the count who can do it for a chip the dominant vendor's tools have never seen, which is small, slow to grow, and about to be asked to do something impossible.
The gap between forty-one and eighty is not an anecdote, and in 2025 it gets measured properly for the first time. The measurement is deliberately unglamorous: a controlled testbed of heterogeneous production accelerators, the same workloads run on each, the default vendor configuration compared against the best a team of experts can reach given effectively unlimited time. Eight campaigns, ninety-two days, 847 configuration-workload pairings per architecture.5Measurement protocol: controlled testbed of heterogeneous production accelerators, eight campaigns, ninety-two days, 847 configuration-workload pairings per architecture. The median gap is 57.3 percent, 95% CI [55.1, 59.4], right-skewed. [measured] The median gap between default and expert-tuned throughput is 57.3 percent, with a 95 percent confidence interval of 55.1 to 59.4, and the distribution is right-skewed, which means the median understates the worst cases, the newest and strangest silicon where the gap yawns widest. The report names it the Configuration Tax: the directly measured, within-architecture gap between default and expert-tuned configuration on the same chip and workload.
The direct Configuration Tax is one layer of a deeper waste, and the two should never be confused. The 57.3 percent is the within-architecture gap between default and optimal configuration on a given chip and workload. Cross-class mismatch is a separate, currently unmeasured category. Kernel inefficiency, cluster fragmentation, and temporal idleness are separate losses again; together, they are not the Configuration Tax.6Production AI fleet utilization below 30 percent of theoretical peak, consistent across published cluster studies. The 57.3 percent substrate-mismatch component and the sub-30-percent total fleet utilization are distinct measurements of distinct losses. [verified-secondary] The two figures, the measured 57.3 percent and the broader sub-third utilization, are different measurements of different things, and the report keeps them apart on every page.
This is happening at the center of the largest capital mobilization in the history of technology. Hundreds of billions of dollars a year, rising toward a cumulative five trillion by 2030, are being pointed at compute hardware that runs at a fraction of its rated capability.7Cumulative AI data-center capex approaching $5 trillion by 2030 on widely cited estimates; annual capex running to hundreds of billions. [verified-secondary] The waste is invisible precisely because it is so widely distributed and so quietly absorbed. Consider the arithmetic from inside a data-center operator: a GPU depreciates whether or not it is used, and whether or not it is used well. A fleet running at thirty percent of potential is a fleet on which two-thirds of the most expensive line item on the balance sheet is evaporating, on schedule, into heat and idle cycles. No operator reports this, because reporting it would mean publishing the fact that the assets the market values them on are idling, and the market has not yet learned to ask.8No operator reports fleet utilization shortfalls because doing so would mean publishing the fact that the assets the market values them on are idling. [thesis / observation] It is eight thousand Priyas, each losing a Monday, summed across an industry and rounded off as a cost of doing business.
The obvious objection, in 2025, is that this is what compilers and autotuners are for, and that the gap will close on its own as the tooling matures. It will not, and the reason is the shape of the problem. An autotuner searches the configuration space of one chip running one workload; it is a per-chip, per-workload effort that yields an answer good until the chip or the workload changes, at which point the search begins again from nothing.9An autotuner is a per-chip, per-workload search; it yields an answer good until the chip or workload changes, at which point the search begins again from nothing. It does not generalize. [thesis] It does not generalize, because it learns nothing transferable; it is a flashlight, not a map. And the thing the autotuner is racing is not standing still. The number of chips is about to explode, and a method that must restart its search for every new chip falls further behind the faster the world moves. The tooling is not the answer. The tooling is the thing the answer has to replace.
The reason the waste does not simply get fixed is that fixing it is human work, and the humans do not scale. The specialist pool grows at roughly six percent a year, the historical rate of an adjacent technical discipline, and it cannot be accelerated with money, because it takes three to five years to train one of these people and the pipeline is fixed.10Specialist pool grows at ~6 percent per year, the historical rate of an adjacent technical discipline; inelastic to capital because the constraint is training time (3-5 years per specialist), not wages. [internal estimate] For seventy years this did not matter, because the number of architectures that mattered was small and stable, anchored by one dominant instruction set and one dominant accelerator vendor. The whole industry made an implicit bet, so deep it was never stated aloud: that hardware diversity is bounded, and that human expertise scales with it. Both halves of that bet are about to come due at once, and the report's central claim is that they come due on a schedule, not a maybe.
And in a small number of research groups, a different idea is taking shape, drawn from an old theorem. In 1970 two cyberneticists proved the Good Regulator Theorem - that every good regulator of a system must contain a model of that system; you cannot reliably control what you have not, in some formal sense, modeled.11Conant-Ashby theorem (1970): every good regulator of a system must contain a model of that system. The corollary for compute: the regulator of all silicon must be a model of how all silicon behaves. [verified-secondary] The corollary, in 2025, is uncomfortable and clarifying: if the system to be regulated is the behavior of all silicon, then the regulator must be a model of how all silicon behaves. Not a faster autotuner, not a better heuristic, but a learned model of the physics underneath the chips, the thing the autotuner conspicuously is not. One such group has a forty-seven-billion-parameter graph neural network in training, fed on the execution traces of hundreds of distinct processors, learning to predict how a chip it is shown will behave on a workload it is given.12HPM-1: the Hardware Performance Model, trained on execution traces of hundreds of distinct processors. CWM-1 is the Compute World Model that uses those profiles to rehearse fleet behavior. In 2025 the stack is a research artifact. [internal / primary research] They call the hardware-performance profiler HPM-1. Its outputs feed CWM-1, the Compute World Model used to rehearse fleet behavior. In 2025 the stack is a research artifact. It does not work yet, and most of the industry has never heard of it.
The markets, in 2025, are not interested. They are pricing the chip makers, the model labs, and the data-center builders, and the layer that makes the chips usable is not yet a category anyone trades. The government is not interested either; its attention is on who manufactures the silicon and who is allowed to buy it, a contest fought one layer below the one that will come to matter. The crisis is measured, enormous, and entirely unaddressed, which is the condition from which every other chapter in this report departs.