Over the past two years, large models have consumed vast amounts of computing power, with training long accounting for the bulk of demand. More recently, the balance of compute demand has been shifting toward inference. Inference computing power needs are quickly catching up with training, while GPU architectures are running into a bottleneck on the inference side that cannot be solved simply by piling on more compute. Data has to move too far, and speed cannot keep up. Among a series of attempts to work around this constraint, SRAM is emerging as one of the leading approaches.

Table of Contents

What New Problems Do Chips Face After Inference Computing Power Demand Overtakes Training?

How Do Groq, Cerebras and Fractile Reduce Data Movement Along Three Different Paths?...

What Shortcomings Will SRAM’s Capacity Ceiling and Process-Node Stagnation Leave Behind?...

What New Problems Do Chips Face After Inference Computing Power Demand Overtakes Training?

As large models move into scaled adoption and routine use, the center of gravity in AI computing power demand is shifting toward inference. Agents, multi-turn conversations and enterprise applications are further multiplying model calls, making inference a primary source of pressure on AI compute costs and infrastructure expansion. [1-1] [1-2]

① SemiAnalysis estimated in its GTC 2026 report that the data center inference market is worth about $50 billion, and that inference already accounts for a larger share of total AI computing power consumption than training, with that share still rising. In agent scenarios, a single request can trigger a chain of model calls more than a dozen times, multiplying inference computing power demand.

② In an analysis published in early 2026, CFA UK assessed the issue through three lenses: growth in AI job demand, the density of enterprise AI deployment, and unit economics for inference. It concluded that inference-side computing power consumption is becoming the dominant part of the model lifecycle.

After inference demand rises, the core bottleneck facing existing chip architectures is not simply peak compute, but data movement during decoding. This bottleneck is commonly known as the “memory wall.” Each time a model generates a token, it must repeatedly read weights and cache data, and memory bandwidth and data travel distance begin to limit inference latency. [1-3]

① Model inference mainly consists of two stages: prefill and decoding. Prefill processes the entire input at once, using matrix-matrix multiplication and leaning compute-intensive. Decoding generates tokens serially, one by one, degrading into matrix-vector multiplication and leaning memory-bandwidth-intensive.

② During the decoding stage, actual computation accounts for less than 20% of latency, while more than 80% is spent on the physical transfer of memory data. Over the past roughly 30 years, processor compute performance has improved by about 50,000 times, while memory bandwidth has grown by about 1,000 times.

Around the memory wall, chip and system research in recent years has increasingly focused on shortening data movement distance. SRAM has drawn renewed attention because it offers on-chip storage, low latency and low energy consumption. [1-4] [1-5] [1-6] [1-7]

① As high-speed on-chip storage, SRAM’s role is to place model weights closer to compute cores, reducing repeated movement of weights and intermediate data between external memory and compute units.

② Studies represented by 2026’s “Memory Wall is not gone,” 2024’s “Memory Is All You Need,” and related NVIDIA modeling all point to on-chip storage area, energy consumption and memory bandwidth as bottlenecks in the decoding stage. Even when computation is pushed closer to storage, on-chip storage systems such as SRAM still become a new constraint.

③ In the decoding stage, HBM4 (High Bandwidth Memory 4) offers bandwidth of about 22 TB/s, while on-chip SRAM reaches 150 TB/s, a roughly 7-fold gap that stems from their physical placement: HBM sits outside the chip, while SRAM is integrated near the compute cores. In terms of energy consumption, HBM is about 20 pJ/bit, while SRAM is only 0.03-0.6 pJ/bit.

How Can SRAM Help Solve the GPU Shortage?

SRAM (Static Random-Access Memory) entered computer systems in the 1960s and has long been used for high-speed temporary storage, cache and on-chip buffering. As large-model inference has taken off, the decoding stage repeatedly reads weights and cache data, bringing SRAM’s low latency, on-chip integration and high bandwidth back into focus as a hardware option for easing the memory wall. [1-14] [1-8] [1-9]

① According to the Computer History Museum, Robert Norman proposed a semiconductor static RAM design at Fairchild as early as 1963. By the late 1960s, bipolar RAM had begun to be used for high-speed scratchpad and cache memory.

② Intel’s first product, the 3101 launched in 1969, was an SRAM chip designed to replace magnetic-core memory modules. Today, SRAM is still mainly used for CPU cache, on-chip storage and small high-speed buffers.

To ease the memory wall, SRAM-based engineering efforts can be broadly grouped into three paths: rearranging data flows at the compiler layer, expanding on-chip storage capacity at the wafer level, and advancing compute-in-memory at the transistor level. All three aim to shorten data movement distance, but they differ in engineering difficulty, maturity and applicable scenarios....