AI Chip Shortage Continues, But There is a Glimmer of Hope

As the adoption of generative artificial intelligence (genAI) continues to soar, not only GPUs are in high demand, but so are the high-performance memory chips needed for AI applications. The current supply in both markets is tight, but the situation is expected to ease.

According to IDC research, 66% of global companies said they would invest in genAI in the next 18 months. Among organizations indicating genAI will see increased IT spending in 2024, infrastructure will account for 46% of total spending. The problem: The critical hardware needed to build AI infrastructure is in short supply.

The rapid growth of AI applications over the past two years has strained the industry’s ability to supply the special high-performance chips needed to run genAI and other AI process-intensive operations. Benjamin Lee, a professor in the Department of Computer and Information Sciences at the University of Pennsylvania, said that most of the focus on processor shortages has been on the exploding demand for Nvidia GPUs and alternatives from various chip designers such as AMD, Intel, and the hyperscale datacenter operators. However, little attention has been paid to the explosive growth in demand for high-bandwidth memory chips, which are fabricated by South Korea’s SK Hynix.

Earlier, SK Hynix stated that due to strong demand, high-bandwidth memory (HBM) products that can be used in combination with high-performance GPUs to handle AI processing requirements are almost fully booked through 2025. HBM prices have also risen 5% to 10% recently due to significant premiums and increased capacity needs for AI chips, according to market research firm TrendForce.

According to Avril Wu, senior vice president of research at TrendForce, HBM chips are expected to account for more than 20% of the total DRAM market value starting in 2024 and may exceed 30% by 2025. Since not all major suppliers have passed customer qualification certification for [High Performance HBM], buyers will accept higher prices to ensure stable and high-quality supply.

HBMs are what supply GPUs with the data they process. Without HBM chips, the memory systems of data center servers would not be able to keep up with a high-performance processor such as a GPU. Anyone buying a GPU for AI computing will also need high-bandwidth memory. Although SK Hynix’s supply is close to its limit, market demand should be able to be met as Samsung and Micron are expanding HBM production.

The current HBM shortages are primarily in the packaging from TSMC (i.e., chip-on-wafer-on-substrate or CoWoS). TSMC is expanding its CoWoS production capacity and expects that supply constraints may ease by the end of next year. According to TrendForce, Nvidia is addressing the GPU supply shortage by increasing its CoWoS and HBM production capacity. This proactive approach is expected to halve the current average delivery time from 40 weeks in the second quarter of 2024 as new production capacity comes online. This expansion is intended to alleviate supply chain bottlenecks hampering the supply of AI servers due to GPU shortages.

Global spending on AI chips is expected to hit $53 billion this year and more than double over the next four years, according to Gartner research. As a result, chipmakers are rolling out new processors as quickly as possible. However, without HBM, these processors may struggle to meet the high-performance demands of genAI.