Running the search

In the Tuning Phase, every call is an experiment: it’s served by a model under test, graded, and counted toward that Step’s search. Run the Workflow the way it will really be used — a test suite, last week’s traffic replayed, a staging environment — and each Step settles on the cheapest model that holds its quality.

SHIMMY_PHASE=tuning python run_eval_set.py

Baseline or Discovery

How a Step is searched depends on what you send as the model:

// Baseline search: pin the model you'd ship today.
openai.chat.completions.create({ model: 'gpt-5.5', messages });

// Discovery search: no Baseline — Shimmy anchors on a frontier model.
openai.chat.completions.create({ model: 'discover', messages });

Baseline search — you pinned a model. It is the quality reference, and only cheaper models are tried against it. Every candidate’s answer is graded against the Baseline’s on the same live input; a candidate that keeps matching it becomes the winner, and the cheapest such candidate wins. If no cheaper model holds up, the Step settles on its Baseline — “keep what you have” is a real answer, and the Report says so.

Discovery search — "discover", no pin. The Step’s first call anchors on a frontier model (within your allowed providers), and later calls run every surviving candidate against the same input, plus the anchor again: every candidate’s answer is graded against the anchor’s answer to that call. Each round halves the field by cost against how surely each one held up; a Step settles when one survivor has passed across enough rounds (three, at 90%, by default). Discovery is faster to settle and better at finding a model you’d never have tried, and costs more per run while it searches.

You can also ask per Step, whatever the request’s model: step(..., mode="discover") searches that Step, mode="off" never searches it.

What the search may try

Your account’s Mode and Allowed providers (dashboard → Settings, or config as code) bound every search:

Tuning runs on Shimmy’s provider keys, so you can try a provider you have no account with. A model offered by several hosts is tried once; the hosts compete on price.

When a Step settles

A settled Step stops being searched; its winner and Backup chain are fixed in the Report. A Step re-opens on its own when:

• its request changes shape — you edited the prompt, added a tool, turned on JSON mode. The settled model keeps serving while the search re-runs against the new prompt, and the Report gets a new version if the answer changes.
• it drifts in Production — see managed Production.

The Workflow is Report ready when every Expected Step has settled. A Step that only runs on a rare path (an error handler, an escalation) can hold the Report up; exclude it on the Workflow’s Steps tab, or finalize the Report early.

Speeding it up

• Report outcomes. Every shimmy.report(schema_valid=True) is evidence the search would otherwise pay to grade. See report outcomes.
• Declare kind. step("classify", kind="classification") tells the search what the Step is for, so it starts in the right place.
• Optimization runs. Settings → advanced search can try several candidates per call for a few runs — fewer runs to settle, more spent per run.

What it costs

Each Tuning call is charged to your prepaid Wallet at what the providers charge Shimmy, plus a margin — grading and the candidates’ calls included. Nothing is charged for a call that ran on your own key.

See billing for prices.