目录
The Request Is Over—Why Does This KV Still Matter?
Before Buying an SSD, Determine How Many GPU Seconds It Saves
Which Operating Parameters Produce the 4TB Estimate?
What Workloads Would Require 1TB, 4TB, and 16TB?
Once Capacity Is Sized, Recovery Performance Determines Which SSDs Are Viable
One High-Capacity SSD Will Typically Serve Multiple GPUs
Four Haircuts Between 4TB/GPU and NAND Orders
Who Will Recognize Revenue First
Three Numbers Will Determine Whether the Economics Work
本内容基于公开资料和研报数据整理，不构成任何投资建议，不代表任何个人观点，仅供学习参考，请理性阅读
The short answer: light reuse implies roughly 1TB/GPU, the base case about 4TB, and only the combination of high concurrency, long retention, and large KV footprints approaches 16TB. But capacity sizing is not a procurement case. Architectural demand translates into enterprise SSD orders only when the GPU seconds saved per TB of cache exceed the SSD’s fully loaded cost.
The Request Is Over—Why Does This KV Still Matter?
For completed sessions, only KV states likely to be reused should move to SSD. A customer may resume a long conversation, an Agent may restart from its previous tool step, or a platform may repeatedly process the same long prefix. Each scenario could avoid a full Prefill pass.
Consider a paused Agent task. It has already read a long document and invoked several tools but must wait for human approval or an external system. When work resumes several hours later, the platform can either recompute the entire context or retrieve the previous KV state from SSD. The former consumes GPU time; the latter consumes storage capacity and recovery bandwidth. Every calculation in this report compares those two costs.
Prefill is the stage in which a model first ingests context and computes attention states. Long documents, extended conversations, and complex system prompts increase this workload. KV cache preserves previously computed historical states, allowing the model to resume directly. Bernstein’s technical analysis of AI memory shows that KV capacity scales with token count and concurrent users. When one Agent passes its output to another, the accumulated context grows further.
One-off queries, anonymous requests that cannot be resumed, and sessions about to migrate to another model have little retention value. Changes to model weights, the tokenizer, positional encoding, or parallel-sharding configuration often make old KV states unusable. Multi-tenant authorization and privacy requirements further limit reuse across users.
For system operators, the first screen consists of four questions: Is the KV state still usable? How soon might it be retrieved? How much compute could it save? Do permissions allow recovery? Only when all four answers are sufficiently clear should the cache become a persistence candidate.
Which Caches Should Be Retained After Leaving HBM?
Before Buying an SSD, Determine How Many GPU Seconds It Saves
Persistence is economical only when the GPU time saved after recovery exceeds the SSD’s fully loaded cost. Low capacity prices alone do not establish the case. If a cache remains stored for 25 days but is never recovered, its write and capacity costs are entirely wasted.
The system must measure “effective hits.” A nominal hit only means that a key was found. An effective hit additionally requires compatible model and parallel configurations, successful authorization, unexpired data, and recovery that is faster than rerunning Prefill. If any condition fails, the hit produces no compute savings.
Expected savings = new persistent-cache sessions per day × 365 × effective hits per session × net GPU seconds saved per hit ÷ 3600 × value per GPU-hour
“Net GPU seconds saved” deducts SSD reads, network transfer, queuing, decompression, and reconstruction time. It therefore better reflects economic value than gross Prefill time. SSD cost should likewise include devices, controllers, networking, power, maintenance, and failure management—not merely raw NAND prices.
The following table illustrates the breakeven point using purely hypothetical parameters. It assumes a fully loaded annual SSD cost of US$250/TB-year and a GPU-hour value of US$3. These are modeling assumptions, not market quotations or any customer’s procurement costs.
In the base case, if each new persistent session generates two effective hits on average, each hit needs to save only about 4.1 net seconds to reach the table’s breakeven threshold. If recovery consumes most of the Prefill savings, the threshold rises rapidly.
A single-point assumption can obscure pricing effects. The table below fixes the base case at 4TB and 400 new cached sessions per day, while varying fully loaded annual SSD cost from US$150—400/TB-year and GPU-hour value from US$2—6. These remain stress-test parameters rather than market quotations.
The table shows the required “effective hits × net seconds saved per hit” for each new cached session. Higher SSD costs raise the retention threshold, while more valuable GPU time allows even shorter Prefill savings to cover capacity costs. Procurement decisions must therefore incorporate current storage costs, GPU opportunity costs, and observed effective-hit data.
