When building AI software, every API call to OpenAI, Anthropic, or Google costs real money. If your application relies on multi-agent execution loops, a single user click can easily trigger 10 to 20 separate model calls.
Without a deliberate caching strategy, your cloud bill will quickly erode your unit economics.
This research note breaks down how we structured semantic caching inside Exogram.ai to cut token spend by over 50% while dramatically improving response speed for end users.
The Semantic Caching Architecture
Traditional web caching matches exact URLs or exact database keys, but AI users ask questions using different words for the exact same intent.
Semantic caching generates a vector representation of the incoming query and checks whether a mathematically similar query already exists inside the cache database.
Step-by-Step Implementation Strategy
1. Set Conservative Similarity Thresholds
Cosine similarity scores range from 0.0 to 1.0.
0.95+ Threshold: Extremely safe for factual, transactional queries, carrying virtually zero risk of serving an irrelevant answer.
0.88 - 0.92 Threshold: Good for general intent classification, though it requires ongoing monitoring for subtle nuance loss.
Rule: Start at 0.94 and tune downward based on real user feedback.
2. Time-to-Live (TTL) Eviction Policies
AI context changes over time, so you should never store semantic cache entries indefinitely.
Static Context (System Rules, FAQs): 30-day TTL.
Dynamic Context (User Career Profiles, Active Workflows): 1-hour to 24-hour TTL.
Real-time State Checks: Do not cache, forcing live deterministic evaluations every time.
3. Edge Rate Limiting and Bot Defense
During an automated attack or bot scrape, malicious traffic can trigger thousands of LLM API calls per minute. Placing Cloudflare Turnstile and strict edge rate-limiting in front of your caching layer ensures that bot traffic is dropped before touching either your cache database or your LLM endpoints.
The Economic Math
By deploying semantic caching and edge pre-filtering across our runtime:
Average Cache Hit Rate: Settled at roughly 38% across routine agent execution loops.
Cost per 1,000 Requests: Dropped from $4.20 down to $1.85.
User-Perceived Speed: Cache hits feel instantaneous, consistently returning under 20ms.
This economic buffer is what allows us to run infrastructure for Exogram.ai and CareerWin.ai efficiently without sacrificing capability or performance.
Continue Exploring from The AI Economist
Built In: Is Anything Standing Between Your AI Agent and Your Database? (https://builtin.com/articles/ai-agent-security-gates)
Systems Infrastructure: Learn how Exogram.ai enforces runtime security and how CareerWin.ai applies context systems to career intelligence.