Captured from the live deployment, not a mockup
01 · The problem
Teams pay full price to answer the same question reworded. Built a drop-in proxy that keeps the OpenAI request and response shape including streaming, verified against the official Python and Node SDKs and LangChain, so adopting it costs one base-URL change.
02 · How it works
- 01
Routed six providers behind one interface from each host's live model list, with the cache as an in-process numpy matrix by default and Redis optional, so it runs with nothing else installed.
- 02
Made it fail open: if the store or the embedder cannot start, the proxy forwards upstream and reports itself degraded rather than failing the request. Measured 77% hit rate and 78% lower cost.
- 03
Matched in two tiers: a normalised hash answers literal repeats with no model call, then a cosine tier over unit-normalised embeddings catches rewording, with the safe similarity threshold measured per embedding model instead of assumed.
- 04
Reached a 77% hit rate and 78% lower cost with zero wrong answers on genuinely new questions, by hashing model, system prompt, temperature, top_p and stop sequences into a namespace and never caching a reply truncated by max_tokens.
03 · What it cost, and what it returned
Built a drop-in semantic cache keeping the OpenAI request and response shape including streaming, routing each model to the host that serves it across OpenAI, Gemini, Groq, Claude, Grok, Ollama and Bedrock.