Not another gateway.
LiteLLM, Portkey and Cloudflare AI Gateway route production traffic, and do it well. Shimmy does its work before you ship — finding the model each step should run on — and can stay out of your production path entirely. Here’s an honest look at what each does.
The capabilities, side by side.
✓ = built in · ~ = partial / basic · — = not offered. We try hard to keep the competitor columns accurate; if we got one wrong, tell us.
| Capability | Shimmy | LiteLLM | Portkey | Cloudflare AI Gateway |
|---|---|---|---|---|
| Wraps your existing OpenAI / Anthropic client | ✓ | ✓ | ✓ | ✓ |
| Recorded responses replayed in tests (offline, no spend) | ✓ | ~ | — | — |
| Per-step model search on your real inputs | ✓ | — | — | — |
| Grades each candidate’s actual answer against your baseline’s answer to the same input | ✓ | ~ | — | — |
| Every candidate answer graded during the search — not a sample | ✓ | — | — | — |
| Tool calls verified (declared tool + valid args), then graded | ✓ | — | — | — |
| Open-weights migration (search only commercially licensed open models) | ✓ | — | — | — |
| A report of every model tried — cost, grades, answers | ✓ | — | — | — |
| Lockfile you commit; production without a proxy in the path | ✓ | — | — | — |
| Fallback across providers on outages and rate limits | ✓ | ✓ | ✓ | ~ |
| Quality drift confirmed against a live baseline, with automatic failover and re-search | ✓ | ~ | — | — |
| Response caching | ✓ | ✓ | ✓ | ✓ |
| Spend caps and alerts | ✓ | ✓ | ✓ | ~ |
| Request logging and analytics dashboards | ~ | ✓ | ✓ | ✓ |
| Self-hosted VPC deployment | ✓ | ✓ | ~ | — |
| Open source | — | ✓ | ~ | — |
| Pricing model | Dev free · Tuning at provider cost + 25% · managed production $4.99/workflow/mo, passthrough free | Free, open source (enterprise plan available) | Subscription + usage tiers | Per-request / Cloudflare plan |
Graded, not predicted.
The question every model-picker has to answer: is the cheaper model’s answer good enough? Here is how each one answers it, from their own documentation.
| Product | How it decides quality | Grades the actual answer against your baseline | Acts on quality drift |
|---|---|---|---|
| Shimmy | Runs the candidate and grades its actual answer against your Baseline’s answer to the same input — every candidate answer during Tuning, then live re-checks in managed Production. | ✓ | ✓ |
| Martian | A trained router predicts the best model for each prompt. | — | — |
| Not Diamond | A ranking model, trained on evaluation scores you supply, picks a model per prompt. | — | — |
| OpenRouter Auto Router | Classifies the task and routes on what is used for it across OpenRouter. | — | — |
| AWS Bedrock Intelligent Prompt Routing | Predicts each model’s response quality and routes to the cheaper one within a quality threshold. | — | — |
| LiteLLM shadow evaluations | On a sample of live traffic, an LLM judge compares the router’s pick with a fixed baseline and reports win and tie rates; acting on them is up to you. | ~ | ~ |
As documented by each vendor, October 2026 — product names link to the source. If we’ve described yours wrong, tell us and we’ll fix it.
When to pick them — and how they fit with Shimmy.
LiteLLM
An excellent open-source gateway and SDK for calling many providers through one interface, with routing, fallbacks, budgets and caching. It routes the models you configure; it doesn’t search for which model each step of your agent should use, or show you why. The two fit together: tune with Shimmy, then put the winning models in your LiteLLM config.
Portkey
A strong production gateway with observability — logs, analytics, guardrails, prompt management. If what you need is to see and govern live traffic, Portkey is solid. Shimmy answers an earlier question: which model each step should run on in the first place, measured on your inputs before launch.
Cloudflare AI Gateway
Caching, rate limiting and analytics at the edge, close to your users. It’s infrastructure for traffic you already route; it doesn’t choose models. Complementary: Shimmy’s passthrough calls whatever endpoint your client points at, Cloudflare included.
Model routers
Martian, Not Diamond, OpenRouter’s Auto Router and Bedrock’s prompt routing pick a model for every production request, in milliseconds — so they predict quality from the prompt rather than grade the answer. That is the right trade-off at request time. Shimmy works earlier, when grading every answer is affordable, and settles each step on a measured winner you can read and commit.
Every model it tried, and every answer it got.
Run one tuning pass and read the Report: the winner for each step, what it cost, and the answers behind the verdict.