4 types of LLM caching, clearly explained. (KV, prefix, prompt, and...
Akshay 🚀@akshay_pachaar
11 views
Oct 24, 2026
~2 min read
Advertisement
1
4 types of LLM caching, clearly explained.
(KV, prefix, prompt, and semantic caching)
An agent can send the same system instructions, tool definitions, and conversation history on every turn. Without reuse, the model processes that shared context again.
That initial prompt processing is called prefill. It contributes to the wait before the first token appears.
The four caches in the graphic reduce repeated work at different points in this process.
𝟭) 𝗞𝗩 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝘀𝗮𝘃𝗲𝘀 𝘄𝗼𝗿𝗸 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝘁𝗼𝗸𝗲𝗻𝘀.
During generation, the model stores keys and values, the vectors attention uses to retrieve information from earlier tokens. Each new token reuses those vectors and adds its own, avoiding repeated computation at the cost of memory.
𝟮) 𝗣𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝘀𝗮𝘃𝗲𝘀 𝘄𝗼𝗿𝗸 𝗮𝗰𝗿𝗼𝘀𝘀 𝗿𝗲𝗾𝘂𝗲𝘀𝘁𝘀.
When a request begins with an already processed sequence, the server can reuse its saved attention state and process the remaining suffix. This helps when requests share instructions, documents, or conversation history.
In vLLM, each cached block’s identifier incorporates the preceding prefix, so identical text after a changed prefix is insufficient for reuse.
𝟯) 𝗣𝗿𝗼𝗺𝗽𝘁 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗽𝘂𝘁𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿 𝗶𝗻 𝗰𝗵𝗮𝗿𝗴𝗲.
A provider manages this prefix reuse through its API, including cache lifetimes and billing rules. Eligible repeated input receives discounted pricing, while the model still generates a fresh answer.
Put stable instructions and documents at the beginning, followed by the changing question, to preserve that reusable prefix.
𝟰) 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗿𝗲𝘂𝘀𝗲𝘀 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿 𝗶𝘁𝘀𝗲𝗹𝗳.
The system converts a prompt into an embedding, a numerical representation of its meaning, and searches for similar stored prompts. An accepted match returns the saved answer without calling the generation model, saving output tokens too.
This suits repeated questions with stable answers, but similarity alone cannot establish that an answer applies to the current user or situation.
These techniques can work together. A semantic cache miss can still benefit from a cached prompt prefix, then use KV caching during generation.
Prefix caching and provider prompt caching share the same underlying reuse. Semantic caching adds a different decision, whether a previous answer is still valid for this request.
That said, I also wrote a full breakdown of KV caching. The article is quoted below.
Hope you enjoyed reading.
Cheers! :)
(KV, prefix, prompt, and semantic caching)
An agent can send the same system instructions, tool definitions, and conversation history on every turn. Without reuse, the model processes that shared context again.
That initial prompt processing is called prefill. It contributes to the wait before the first token appears.
The four caches in the graphic reduce repeated work at different points in this process.
𝟭) 𝗞𝗩 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝘀𝗮𝘃𝗲𝘀 𝘄𝗼𝗿𝗸 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝘁𝗼𝗸𝗲𝗻𝘀.
During generation, the model stores keys and values, the vectors attention uses to retrieve information from earlier tokens. Each new token reuses those vectors and adds its own, avoiding repeated computation at the cost of memory.
𝟮) 𝗣𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝘀𝗮𝘃𝗲𝘀 𝘄𝗼𝗿𝗸 𝗮𝗰𝗿𝗼𝘀𝘀 𝗿𝗲𝗾𝘂𝗲𝘀𝘁𝘀.
When a request begins with an already processed sequence, the server can reuse its saved attention state and process the remaining suffix. This helps when requests share instructions, documents, or conversation history.
In vLLM, each cached block’s identifier incorporates the preceding prefix, so identical text after a changed prefix is insufficient for reuse.
𝟯) 𝗣𝗿𝗼𝗺𝗽𝘁 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗽𝘂𝘁𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿 𝗶𝗻 𝗰𝗵𝗮𝗿𝗴𝗲.
A provider manages this prefix reuse through its API, including cache lifetimes and billing rules. Eligible repeated input receives discounted pricing, while the model still generates a fresh answer.
Put stable instructions and documents at the beginning, followed by the changing question, to preserve that reusable prefix.
𝟰) 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗿𝗲𝘂𝘀𝗲𝘀 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿 𝗶𝘁𝘀𝗲𝗹𝗳.
The system converts a prompt into an embedding, a numerical representation of its meaning, and searches for similar stored prompts. An accepted match returns the saved answer without calling the generation model, saving output tokens too.
This suits repeated questions with stable answers, but similarity alone cannot establish that an answer applies to the current user or situation.
These techniques can work together. A semantic cache miss can still benefit from a cached prompt prefix, then use KV caching during generation.
Prefix caching and provider prompt caching share the same underlying reuse. Semantic caching adds a different decision, whether a previous answer is still valid for this request.
That said, I also wrote a full breakdown of KV caching. The article is quoted below.
Hope you enjoyed reading.
Cheers! :)
