Abstract

Transformer-based large language models (LLMs) have already achievedremarkable results on long-text tasks, but the limited GPU memory (VRAM)resources struggle to accommodate the linearly growing demand for key-value(KV) cache as the sequence length increases, which has become a bottleneck forthe application of LLMs on long sequences. Existing KV cache compressionmethods include eviction, merging, or quantization of the KV cache to reduceits size. However, compression results in irreversible information forgetting,potentially affecting the accuracy of subsequent decoding. In this paper, wepropose SpeCache, which takes full advantage of the large and easily expandableCPU memory to offload the complete KV cache, and dynamically fetches KV pairsback in each decoding step based on their importance measured by low-bit KVcache copy in VRAM. To avoid inference latency caused by CPU-GPU communication,SpeCache speculatively predicts the KV pairs that the next token might attendto, allowing us to prefetch them before the next decoding step which enablesparallelization of prefetching and computation. Experiments on LongBench andNeedle-in-a-Haystack benchmarks verify that SpeCache effectively reduces VRAMusage while avoiding information forgetting for long sequences withoutre-training, even with a 10x high KV cache compression ratio.