Chasing the Frontier or Letting the Chips Fall Where They May? 

If the world relies on a single Taiwanese company for making 3nm chips, it now also relies on two South Korean companies on memory chips crucial to the artificial intelligence (AI) boom (see Figure 1). Indeed, riding the AI wave has only entrenched South Korea’s monopoly on high-bandwidth memory (HBM) chips that are placed on Nvidia graphics processing units (GPUs) to power AI data centers. In fact, Nvidia alone is expected to spend billions this year to snatch up nearly half of the global HBM supply.

Figure 1. South Korea Monopolizes HBM MarketSource: Goldman Sachs Global Investment Research.

That voracious appetite for HBMs has done well for SK Hynix stock over the last year, as the company pioneered the technology for AMD’s Fiji GPU in 2015 and remains at the frontier. Given the $90 billion of capital expenditure allocated for AI centers globally—the driver of both GPU and HBM demand—it’s an opportunity difficult for chip manufacturers to ignore. This is especially so for Chinese suppliers that, for both self-interest and national-interest reasons, are eager to dive all-in on whatever the frontier chip may be (see Figure 2).

Figure 2. China Ranks Second in Data Center Annual RevenueSource: Statista Market Insights.

Yet HBM’s medium-term prospect may be murky, as uncertainties around technology choices and the AI boom itself mean that HBM’s market could peak sooner than expected. It would likely be prudent for second-movers, such as Chinese suppliers, to evaluate the sustainability of HBM’s growth cycle rather than rushing to develop capacity.

The history of technology cycles is littered with sunk costs, and getting the timing right on big strategic decisions can be determinative of success. For example, Micron missed the boat on HBM because it stuck with its hybrid memory cube until 2018 and is now playing catch-up to the South Koreans.

What’s So Special about HBM?

Most memory chips are commodity-like legacy chips that don’t get much attention. But HBM happens to be a chip that is optimized for the current AI industry focus on training large language models (LLMs). Its main advantage is the ultra-efficient architecture that allows it to transfer data faster over shorter distances on an AI chip (see Figure 3).

Figure 3. HBM Chips Are Essential Components of GPUs
Note: HBM is stacked vertically and directly besides the GPU on a silicon interposer.
Source: Rambus.

HBM’s combination of speed and efficiency means it vastly outperforms the current industry standard, graphics double data rate (GDDR), but it is also 3-5 times more expensive. The higher cost of HBM seems justified at the moment as its properties make it ideal for AI data center applications. Unsurprisingly, growth in AI data centers alone will be responsible for lifting HBM’s market value over the next few years (see Figure 4).

Figure 4. HBM To Surpass GDDR Memory in Market Value by 2025
Source: Goldman Sachs; Verified Market Research.

Short-Term Memory?

The latest frenzy over chips risks being overwhelmed by short-termism. For a Chinese player like Huawei, which now relies on buying Korean HBM chips, it is already spending money to incubate HBM production. That’s a sure invitation for domestic competitors like Biren Technology and Moore Threads to follow suit. Yet according to industry estimates, Chinese memory suppliers are likely behind the HBM frontier by some ten years.

Over the coming decade, the HBM market may see a lower ceiling for growth than the current hyper cycle suggests for two reasons. One, HBM is currently a niche chip used for training AI models in data centers and isn’t likely to become an on-device memory chip inside smartphones and laptops, limiting its longer-term addressable market. Two, even as HBM’s unit cost is expected to fall by 40% as manufacturers scale up production, an emerging universal memory chip that would be as good for AI training as it would be for other computing applications could further stunt HBM’s growth by the end of the decade.

The technology behind this “goldilocks” memory chip is called phase change memory (PCM), which holds the promise of delivering a universal memory chip that combines the low latency of DRAM with the non-volatility of flash memory, the best of both worlds.

PCM isn’t new but was always constrained by its material properties that couldn’t balance speed, efficiency, and durability. That is until material scientists at Stanford University’s Pop Lab of Electrical Engineering recently made what appears to be a major step towards commercializing PCM. They discovered a novel alloy that remained stable while seeing massive performance gains and is compatible with commercial manufacturing processes. Meanwhile, the Korea Advanced Institute of Science and Technology has had some success miniaturizing PCM architecture so that it can fit onto small chipsets.

While a universal memory chip isn’t knocking at HBM’s door just yet, it is worth considering technology cycles that often turn today’s winner into tomorrow’s casualty. Examples are legion. Carbon fiber displaced other metal alloys in modern aircraft; silicon carbide and gallium nitride are replacing power chips in automobiles; and floppy disks gave way to USB flash drives.

Chinese suppliers pride themselves on speed, but in their bid to chase the frontier of chips, that speed may lead to sunk costs as technology cycles move even faster. For late-comers, clearly assessing these rapidly changing markets instead of betting the house on the current new thing could avoid wasting precious time and resources that might be better allocated elsewhere.

AJ Cortese is a senior research associate at MacroPolo. You can find his work on industrial technology, semiconductors, the digital economy, and other topics here.