Questions, answered.

Shimmy finds the best LLM for every step of an AI app or agentic workflow on cost and quality, before you ship it. While you build, it records each step once so tests replay offline for free. While you tune, it searches each step — cheaper models measured against the one you would ship, on your real inputs — until every step settles. You get a Report with the winner for each step, its backups and every model tried, and a shimmy.lock file to ship on.

Because the steps are different jobs. A classifier picking one of five labels, a query rewrite before retrieval, a planner breaking down a request and a writer drafting the reply need very different models, and an app or agent run on one model is either overpaying for its easy steps or underpowered for its hard ones. Shimmy measures each step on its own inputs.

Yes. A chat assistant, a RAG feature or an endpoint that summarizes or extracts is a Workflow like any agent: each LLM call site is a Step, and each request or conversation is a run. An app with two Steps tunes and settles the same way an agent with twenty does, usually faster.

Each candidate answers the same live input as your baseline model, and its actual answer is graded against your baseline’s: how likely it is to be clearly worse, somewhat worse, equivalent, somewhat better or clearly better. It passes a call when it is at least 70% likely to be as good or better and under 15% likely to be clearly worse; tool calls must first name a declared tool with arguments that parse. A step settles on it only after 20 graded calls at 90%. Outcomes your own code reports, like a schema that validated, count too. If nothing cheaper holds, the step keeps your model, and the Report says so.

Jev, a fast decision model from TypeSafe built to return probabilities rather than write text. A grade typically takes about 0.2 seconds and costs a fraction of a cent, which is why every candidate answer during Tuning is graded instead of a sample. Each comparison is asked both ways round — candidate against baseline, and baseline against candidate — to cancel a lean toward either side. If Jev is unavailable, an LLM judge grades on the same scale. A grade is a model’s judgment, not ground truth: outcomes your code reports outrank it, and the Report keeps every answer so you can read them.

Use a Discovery search: send model "discover" and Shimmy anchors the step on a frontier model, races several cheaper candidates per call, and halves the field each round until one has matched across enough rounds. It settles in a few runs.

It depends on how many steps you have and how often each runs: every Tuning call moves its step’s search along. A Discovery search usually settles a step in a handful of runs; a Baseline search needs enough graded calls for each candidate it tries. Reporting outcomes from your code settles steps sooner.

Development is free: recordings replay at no cost, and a free Dev allowance covers making them. Tuning is billed at what the model providers charge plus 25%, from a prepaid wallet. In production, passthrough is free and managed production is $4.99 per workflow per month.

Because what Tuning costs depends on your workflow, and that is hard to know before you start. A prepaid wallet makes the ceiling yours: Tuning spends what you put in and no more, and pauses when it runs out. The Tuning estimator gives a range for sizing the first top-up.

Development is free, including a $5 Dev allowance for making recordings — no card required. Passthrough production is free forever. You pay for Tuning, and for managed production if you want it.

Wallet credits expire 365 days after purchase and are not refundable. The oldest credit is spent first.

No. Export the lockfile and run the SDK in passthrough: it calls your provider directly with each step’s winner, and falls back down that step’s backup chain on an outage, a rate limit or a refusal. Shimmy is not in the path.

Drift detection against a live baseline: a few calls a day are re-sent to your baseline in the background and the winner’s answer is graded against it. One failed check grades the step’s next two calls at once; two failures in three and its backup serves immediately while the step is re-searched. Plus failover across providers, hedged requests for latency-sensitive steps, a response cache, monthly spend caps, and alerts by email or signed webhook.

Each settled step has a backup chain: its winner, the best passing runner-up on a different provider, then your baseline. Both passthrough and managed production walk it, so an outage at one provider fails over to another automatically.

SDKs for Python, TypeScript and Rust, wrapping the OpenAI and Anthropic clients. Searches can span OpenAI, Anthropic, Perplexity, Mistral, Cloudflare and open-weights hosts like Ollama Cloud — limited to the providers you allow. An open-weights migration mode tries only commercially licensed open-weights models.

Tuning runs on Shimmy’s own provider keys, so you need none. For managed production you store your provider keys with us; they are encrypted at rest and never shown again. Shimmy API keys are stored as hashes — the plaintext is shown once.

Tuning keeps them for the life of the Report — they are what the Report shows you, model by model. Outside Tuning, storing content is a setting you can turn off. Sensitive values such as emails, cards and keys are masked in what we store by default.

Yes. Under pytest, vitest, jest, node --test or cargo test, the SDK replays recordings from your repo with no network. In CI, a test whose input no longer matches a recording fails rather than calling a model.

Yes. A self-hosted build ships as a Docker image with Postgres. Traffic and keys never leave your VPC; pricing is a flat enterprise license.

A gateway routes your production traffic; it is in the path of every call, forever. Shimmy does its work before you ship: it finds the right model for each step during development and tuning, and hands you the answer as a file. You can keep it in the path for drift protection — or not.

A router picks a model for every production request in milliseconds, so it has to predict quality from the prompt. Shimmy runs each candidate and grades its actual answer against your own baseline’s answer to the same input, before you ship, and settles each step on a measured winner you can read in the Report and commit as a lockfile. In managed production it keeps re-checking that winner against a live baseline and fails over automatically when quality drifts.

An eval tool scores the models you choose to test, on datasets you assemble. Shimmy chooses the candidates, runs them on your real inputs as your app or agent executes, grades every answer against your own baseline’s answer to the same input, and stops when each step has settled.

A mock returns what you wrote; a recording returns what the model actually said, captured once from a real call. Shimmy also logs every call your code makes, so a test can assert on which steps ran and in what order, and inject the failures providers really produce.