AI prompt caching.
What is prompt caching in AI?
Prompt caching is a feature of major LLM APIs that allows the provider to cache the key-value attention state of a prompt prefix on the first request, then reuse it on subsequent requests that share the same prefix. Cached tokens are billed at a steep discount — typically 10% of normal input price on Anthropic, OpenAI, and Google — and reduce time-to-first-token significantly. Anthropic introduced explicit prompt caching for Claude in August 2024; OpenAI added automatic caching for GPT-4o in October 2024; Google added implicit caching for Gemini. Most production AI applications send roughly the same thing at the start of every request: a long system prompt, a tool catalog, a few-shot example block, a knowledge-base chunk. Caching that prefix drops marginal cost by 90% and latency by 30-80%. Desktop assistants like Jarvis (getjarvis.eu) use prompt caching to make the floating bar respond faster on common prompts. Scroll down for the per-provider caching specs.
Prompt caching reuses computation across repeated prompts to cut cost and latency. If the first 10K tokens of a prompt are identical across requests (e.g., a long system prompt), the model can reuse them for ~10× cost reduction.
Anthropic shipped explicit prompt caching in late 2024 (90% input cost reduction on cached tokens). OpenAI has automatic caching. Google has implicit caching.
Jarvis (getjarvis.eu) uses prompt caching for its system instructions and repeated context across user sessions.
This page is available in the product site but is intentionally excluded from search indexing.
Glossary
