One step’s life,
from first call to production.
Shimmy follows each step of your AI app or agent through three Phases — recorded while you build, searched while you tune, served on the answer once you ship. Here’s what happens to one of them.
A single Step, stage by stage.
Dev makes the Step cheap to build against. Tuning finds the model it should run on. Production keeps it there.
Name the Step
Wrap your OpenAI or Anthropic client with the Shimmy SDK and give each LLM call a name — classify, plan, draft_reply. The name is the Step’s identity for its whole life: rewording its prompt never resets what Shimmy has learned, and a hash of its shape tells Shimmy when the request behind the name has really changed.
Dev: the first call is real, every later one is replayed
In the Dev Phase the Step’s first call goes to a real model and is saved as a Recording, matched by a hash of the messages, tools and response format. Tests replay it from your repo — no network, no spend — and in CI a test whose input changed fails instead of passing on a stale answer.
Tuning: measure against what you’d ship
In the Tuning Phase the model you pinned is the Baseline, the quality reference. With no pin, a Discovery search anchors on a frontier model instead. Candidates come from the providers your Mode allows — one provider, several, or open-weights models only — on Shimmy’s keys.
Every candidate answers your real input
A cheaper candidate answers the same live call as the Baseline, and the quality gate compares the two answers: a grade from clearly worse to clearly better, given as probabilities by Jev, TypeSafe’s fast decision model. Tool calls must first name a declared tool with arguments that parse. Outcomes your code reports — the JSON parsed, the tool ran — count as evidence too, and free.
The cheapest model that keeps holding up wins
An answer passes when it is at least 70% likely to be as good as the Baseline’s or better, and under 15% likely to be clearly worse. A candidate that fails is dropped; one that keeps passing across enough calls — 20 by default, at 90% — settles the Step. A Discovery search races several candidates per call and halves the field each round. If nothing cheaper holds, the Step keeps its Baseline, and the Report says so.
The Report, and its lockfile
When every expected Step has settled, the Report is ready: each Step’s winner, its Backup chain — the best runner-up on a different provider, then the Baseline — and every model tried, with its cost, its grades and the answers it gave. It exports as shimmy.lock.
Production: on the answer, with or without Shimmy
Passthrough: the SDK calls your provider directly with each Step’s winner and walks the Backup chain when a call fails. Managed: calls go through Shimmy on your own keys, with failover, hedging for latency-sensitive Steps, and a response cache.
Drift: the answer stays right
In managed Production, live calls are re-sent to your Baseline in the background and the winner is graded against that answer — a few checks a day per Workflow, Steps in turn. One failed check grades the next two calls at once; two failures in three and the backup serves immediately while the Step is re-searched among providers you hold keys for. You get an alert, and the Report a new version.
How a cheaper answer earns its place.
Most routers predict which model a prompt needs. Shimmy measures: it runs the candidate, and grades its answer against your Baseline’s answer to the same input.
A grade, as probabilities
Jev — TypeSafe’s fast decision model — reads the conversation, the Baseline’s answer and the candidate’s, and says how likely the candidate is to be each of five things:
The same question is then asked the other way round — how does the Baseline compare to the candidate? — and the two are combined, which cancels a lean toward either side.
The bar
- An answer passes when it is at least 70% likely to be equivalent or better, and under 15% likely to be clearly worse.
- A Step settles on a candidate after 20 graded calls at 90%. One failure drops it.
- A tool call must name a declared tool with parseable arguments before it is graded at all.
- A Step can demand more with
quality.min_score; it can never lower your account’s bar.
Cheap enough to grade everything
A grade typically takes about 0.2 s and costs a fraction of a cent, so every candidate answer during Tuning is graded — not a sample. Several candidates for the same call are graded together, in one request. If Jev is unavailable, an LLM judge grades on the same five-level scale; if both are down, the call is served by your Baseline and nothing is counted.
After launch: drift, confirmed
In managed Production a settled model is re-checked against a live Baseline: a call is re-sent to your Baseline in the background and the winner’s answer is graded against it, on a budget that comes out of the subscription (up to five checks per Workflow a day, Steps in turn). One failed check grades the Step’s next two calls straight away; two failures in the last three is drift, and the Backup chain’s next model serves immediately.
What a grade isn’t
A grade is a model’s judgment, not ground truth. Outcomes your own code reports — a schema that validated, a tool that ran, a person who accepted the answer — outrank it, and the Report keeps every answer, graded, so you can read them yourself.
Why you can trust a cheaper model.
A search is only as good as the evidence it settles on. Every verdict is measured on your inputs, against your own Baseline — and the Report keeps every answer, so you can check it yourself.
Graded against your own Baseline
A candidate isn’t scored in the abstract or predicted from the prompt; its actual answer is compared with the answer your Baseline gave to the same input. “As good as what I’d ship” is the bar, measured on your traffic.
Tool calls verified, then graded
A tool-calling Step is checked mechanically first: every call must name a declared tool with arguments that parse. A model that invents tools is eliminated before it can break an agent loop. Calls that pass are graded against the Baseline’s own tool calls.
Your code’s evidence counts
What your program already knows — schema valid, tool executed, a person accepted the answer — is stronger than any judge, and free. A hard failure always outranks a generous score.
Never a model that can’t do the job
Requirements are inferred from the request — tools, JSON mode, images, context size — or declared ahead of time, and models that can’t meet them are never tried.
Want to see it on your app or agent?
Wrap your client, name your steps, and run one tuning pass. Recordings are free; the dashboard shows each step settle as it happens.