Talk to us
Shimmy docs

Report outcomes

The part that pays for itself.

What it replaces

To know whether a cheaper model held up, the quality gate dispatches a second, more expensive call and pays an LLM judge to compare the two. That is the honest cost of finding out, and it reaches you as exploration_cost on every call it grades.

Your program already knows the answer. It parsed the JSON, or it didn’t. The tool executed, or it threw. A person accepted the output, or rewrote it.

Those are not approximations of quality — they are the facts the judge is trying to predict, available for free to anyone standing inside the program, and invisible to a proxy watching HTTP traffic.

report-outcomes.ts 
import OpenAI from 'openai';
import { Shimmy, optimizerMeta } from '@rfa-labs/shimmy';

const shimmy = new Shimmy();
const openai = shimmy.wrap(new OpenAI());

interface Ticket {
  category: string;
  urgency: number;
}

await shimmy.run('triage', async () => {
  await shimmy.step('extract', { kind: 'extraction' }, async () => {
    const res = await openai.chat.completions.create({
      model: 'auto',
      messages: [{ role: 'user', content: 'Extract category and urgency.' }],
      response_format: { type: 'json_object' },
    });

    const text = res.choices[0]?.message.content ?? '';

    // Explicit: you decide what counts as valid.
    try {
      const ticket = JSON.parse(text) as Ticket;
      await shimmy.report({ schema_valid: typeof ticket.category === 'string' });
    } catch {
      await shimmy.report({ schema_valid: false });
    }

    // Or let a throw speak for itself — `verify` reports either way and
    // rethrows, so control flow is unchanged.
    const parsed = await shimmy.verify(() => JSON.parse(text) as Ticket);

    // The accounting the edge returns. `exploration_cost` trends to zero for a
    // step as reported outcomes accumulate.
    const meta = optimizerMeta(res);
    if (meta) {
      console.log(`saved ${meta.saving}, explored ${meta.exploration_cost}`);
    }

    return parsed;
  });

  // A human verdict is the strongest signal of all, and it never crosses the
  // wire on its own — only you can report it.
  await shimmy.report({ human_verdict: 'accepted' }, { stepId: 'extract' });
});
Compiled in CI. Rust's equivalents are on the client reference.

The signals

SignalMeaning
schema_validThe response parsed and validated. The strongest cheap signal there is.
tool_executedThe emitted tool call actually ran.
retriedYou re-ran this step because the output was inadequate.
run_completedThe run reached its terminal state.
human_verdictaccepted, edited, or rejected.
scoreYour own eval score in [0,1].

Report only what you actually know. Every field is optional, and inferring a signal from silence would manufacture evidence.

How they are weighed

A hard failure — schema invalid, tool failed, step retried, human rejected — is decisive. It scores zero and outranks anything else sent alongside it, including a generous score. That asymmetry is deliberate: a client must not be able to report 1.0 next to a failed parse and manufacture safety for a model it could not use.

Otherwise: your explicit score wins, then a human verdict, then a clean structural pass (0.95 — “it parsed and ran” is strong evidence of adequacy, not proof of excellence).

Nothing conclusive reported means inconclusive, and the judge decides exactly as it did before. Absence of evidence is never evidence.

When it starts paying

Reported outcomes accumulate per (step, model) pair. Once a pair clears the same bar the class-level policy uses, the gate stops dispatching a baseline call for it — no second call, no judge.

The response tells you when a step has crossed that line:

const res = await shimmy.report({ schema_valid: true });
res?.judge_free;   // true once this pair serves without grading
res?.samples;      // conclusive reports so far
res?.passes;       // how many cleared the bar

Evidence never transfers between models. Verifying haiku on a step says nothing about sonnet, and pooling them would let one model’s track record authorize another’s downshift.

Automatic reporting

Where “did it work” is simply “did it throw”, let the exception speak:

const parsed = await shimmy.verify(() => JSON.parse(text));

It reports either way and rethrows, so your control flow is unchanged. The LangChain adapter does the same for tool calls automatically — a tool that ran is direct evidence the model’s call was well-formed.

Best-effort by design

Reporting failures are logged, never thrown. A telemetry problem must not become an application problem — the call already succeeded, and the outcome is an observation about it.

Next