DeepSeek's Memory Use: A Short Claim About KV Cache

SUMMARY

In 30 seconds

In a short excerpt, Matthew Berman says DeepSeek's KV cache needs one quarter of the HBM and one eighth of the SSD storage, allowing more tokens to be served with less memory. The clip does not identify the model, baseline, workload or measurement behind those ratios, so they cannot be treated as general performance guarantees.

1Memory supply and costBerman starts by linking constrained supply with rising prices.
2Two storage ratiosHe cites one-quarter HBM and one-eighth SSD storage for a DeepSeek KV cache, but the clip does not specify the baseline.
3Serving more tokensBerman argues that using less memory can enable more tokens to be served. The excerpt gives no measured throughput or test conditions.

Takeaways

Berman connects lower KV-cache memory requirements with the ability to serve more tokens.
He cites one-quarter HBM and one-eighth SSD figures without identifying the comparison in this excerpt.
Actual memory needs and speed depend on the model, workload and implementation.

TopicsKV caching & prefix reuse Inference Infrastructure & deployment

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Memory constraints and cache

Berman discusses constrained memory supply and claims reduced cache storage requirements. The comparison behind the figures is absent from the excerpt.

SLIDE 1 / 1The cache comparison illustratedThe clip depicts Berman's one-quarter HBM claim, without specifying the comparison baseline.

Matthew Berman says constrained supply raises prices, then turns to DeepSeek's KV cache. He claims that the cache needs one quarter of the HBM and one eighth of the SSD storage. The excerpt does not identify the baseline, model or workload used for those figures.

Serving implications

He relates lower memory requirements to serving more tokens and describes the model as fast. No throughput measurement is given.

Berman argues that lower memory requirements allow more tokens to be served without paying for as much memory, and describes the model as fast. The clip does not give a throughput measurement or test conditions; actual savings and speed depend on implementation and workload.