Eight stages,
traced end to end.
The Optimizer sits between your agents and your LLM providers, intercepting each request and applying four optimization techniques in sequence. Here’s what happens to a single call — from your code to the response you get back.
A single request, step by step.
The order is deliberate: the cheapest call is the one we never make (cache) → the one we make smaller (trace + route) → spend extra compute only when quality requires it and stop the moment it’s met (verify-and-stop) → never serve a degraded answer that fails your quality gate → meter baseline vs actual.
Point your base_url at us
Change one line — your base_url — and every call now flows through the Optimizer. We’re OpenAI/Anthropic-compatible, so no code changes. You keep your own provider keys; they ride per-request in the x-provider-key header.
We classify the request
Every request is fingerprinted into a task-class by what it asks for — never by how much data it fetched. The classifier reads only the instructions and the question (system + user turns), not retrieved tool results; “summarize this ticket” is easy whether the ticket is 200 or 20,000 tokens. The class is the difficulty verdict plus the agent-turn flag — no raw prompt text is stored.
Check exact + semantic cache
If we’ve seen this exact prompt before, serve from cache — no upstream call. If a semantic rephrase (same intent, different words) is in cache and cosine similarity ≥ 0.92, serve that. ~30% of calls hit cache on typical workloads.
Route to the cheapest model the evidence trusts
We consult the learned routing evidence for this task-class on two levels: the class policy (proven match-rate over many samples) and the verdicts of the semantically nearest graded prompts — so even a never-seen class warms up on its closest analogs instead of starting from zero. Strong evidence → serve the cheap model directly, no verification call. Uncertain → explore: dispatch the cheap model and verify.
Verify the cheap path (exploration only)
Prose answers are scored by an LLM judge against your rubric — accept at ≥ your threshold (default 0.70), else transparently fall back to your baseline. Tool calls are verified structurally instead (every call must name a declared tool with parseable arguments — no judge needed, and a hallucinated call falls back silently so your agent loop never breaks). Either way: no degradation is ever served to save a dollar.
Record the outcome — the system learns
Every verified attempt is recorded: the class-level aggregate AND a similarity-indexed graded note. As evidence mounts, the class flips to learned-safe and the judge/baseline verification cost amortizes away. Once enough graded outcomes exist, a small trained router learns your workload’s shape and routes new prompts directly — the system literally gets cheaper the longer it runs.
Meter the saving
We compute baseline cost (what your declared model would have cost) minus actual cost (what we served), clamped non-negative. During exploration we paid for both calls, so the exploration saving clamps to ~0 — we never bill you for savings we didn’t make. The saving is attributed to a stage and appended to your auditable ledger.
Return the response
You get a standard OpenAI-shaped completion plus an optimizer block: baseline model, actual model, baseline cost, actual cost, saving, and source. Full transparency. You pay 20% of the saving — if it doesn’t save, it doesn’t cost.
Where the savings come from.
The base gateway (cache + route + fallback) is commoditized by 2026. Our defensible value is the layer on top — three wedges incumbents don’t do, over a memory substrate that makes savings compound.
Semantic response caching
Caches responses by cosine similarity, not exact match. Users ask the same question many ways — “summarize this”, “give me the key points”, “what’s this about” — and all hit the same cached response. Embedding model: fastembed AllMiniLML6V2 (384-dim). Threshold: 0.92–0.97, per-tenant tunable.
Quality-bounded routing
Routes to the cheapest model the evidence trusts. Economy (gpt-4o-mini, claude-haiku), standard, or frontier. Every downshift is verified before it’s trusted: prose outputs are scored by an LLM judge against your rubric; tool calls are verified structurally against your declared tools (a hallucinated call never reaches your agent loop). Below the bar, we fall back transparently. Routing difficulty is judged on what the task asks — never on how much data it fetched — and tool-calling traffic gets a capability floor, never a weak model that would break the loop.
Test-time-compute pruning
Verify-and-stop: spend extra reasoning only where it changes the answer, and stop the moment it’s good enough — the opposite of best-of-N spam. 2026 research: confidence-gated early stop achieves ~74% of best-of-16’s gain at ~1.2 samples.
Agentic plan caching
Caches the workflow structure (plan template), not just responses. Agent outputs depend on external context, so plain response caching fails — but the structure of a workflow is stable. Retrieve by semantic similarity, adapt to current context. The 2026 answer to agent cost.
Optimization memory — why savings compound.
A flat cache remembers responses. The Optimizer remembers judgments, traces, and policies — and links them. Month 3 is cheaper than month 1 at equal quality, because the system has learned your workload. That’s the defensible layer under all three wedges.
Routing policies + routing memory
Two-level learned downshift safety. The class-level policy tracks proven match-rate per task-class, baseline, and model; the routing memory stores every graded exploration as a similarity-indexed note, so a new prompt borrows the verdicts of its nearest analogs. Cold classes warm on their neighbors instead of starting from zero — judge-free serving arrives sooner.
The learned router
Once a tenant has enough graded outcomes, a small model trained on their own evidence starts predicting which prompts are safe to downshift — replacing verification on confidently-easy calls. The flywheel: exploration produces the training data, the trained router then eliminates future exploration cost.
Verification outcomes
Every graded attempt — judge scores on prose, structural verdicts on tool calls — is recorded. Tool-call verification (declared tool + valid args) means agent-heavy traffic learns like prose traffic does, instead of being exempt from learning.
Plan templates + knowledge graph
Workflow structure extracted from completed runs, retrieved by semantic similarity, adapted to current context. Typed layers (routing policy, plan templates, verify results, notes) linked in a graph (related[]); hybrid retrieval: semantic cosine + keyword + one-hop neighbors. Opt-in anonymized cross-tenant learning lets a new tenant’s cold start borrow the network’s accumulated routing knowledge.
Want to see it on your traffic?
Point your base_url at us and watch the dashboard light up with real, attributed savings. $5 free allowance, no credit card.