How Semantic Caching Can Reduce LLM API Costs

The problem

LLM cost grows with request volume and token usage. In production, repeated questions, retries, navigation flows, and slightly different phrasings can cause the same expensive reasoning to happen repeatedly.

How semantic caching works

Instead of using only an exact string as the cache key, embed the incoming request and compare it with embeddings stored for previous requests. If the similarity score crosses a carefully chosen threshold, return the cached response.

user request → embedding → similarity search → cache hit → response
                                      ↓ miss
                                   LLM call → store result

Important design decisions

• Use a threshold validated against real examples rather than an arbitrary similarity value.
• Include model, system-prompt version, retrieval context version, and important parameters in the cache identity.
• Set TTLs and invalidate cached answers when source data changes.
• Measure hit rate, false-hit rate, latency, token savings, and cache storage cost.

When not to cache

Avoid reuse when answers depend on rapidly changing private state, time-sensitive data, user-specific permissions, or prompts where a small semantic difference changes the correct answer.

Production measurement

For my portfolio work, I report approximately 30% lower LLM latency from caching. This is an author-attributed engineering result, not an independently audited benchmark. A production evaluation should report request volume, cache hit rate, model mix, token distribution, and percentile latency.

Takeaway

Semantic caching is not simply “put embeddings in Redis.” It is a correctness problem first and a performance optimization second.