Eval Harness First

---
name: eval-harness-first
description: Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
---

# Eval Harness First

The Phase 0 gate for the whole plugin:
`finetuning-method-selection` and every downstream
skill assume this harness exists before a training
config gets written. The harness is not a run-end
side artifact — it is the data-curation engine. The
same labeled traces that build the goldens feed
training data, minus an explicit holdout.

**Input:** production/agent traces if they exist, or
a task spec if they don't, plus labelers willing to
grade ≥100 examples.
**Output format:** the `eval/` directory below —
goldens, graders, drift suite, and the base-model
baseline that later phases gate on.

## The Gate

No eval harness, no fine-tune. Skip to a training
config and there is nothing to measure against,
nothing to catch regressions, and no labeled data
to train on. The flywheel:

1. **Collect traces** — production/agent spans, or
   synthetic tasks if none exist yet.
2. **Error analysis** — open coding on ≥100 traces,
   axial coding into 4–8 failure buckets.
3. **One grader per bucket** — deterministic first;
   calibrated LLM-judge only for genuinely
   subjective criteria.
4. **Prioritize** by frequency × severity × value.
5. **The labeled traces feed dataset curation, minus
   an explicit holdout.** Every `eval/goldens.jsonl`
   ID stays excluded from training data by ID.
6. **Train.**
7. **Re-run the same harness** on the checkpoint —
   not a different, looser one.
8. **Drift detection feeds back to step 2** — new
   production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it
must exist first — it is both the training data
source and the checkpoint's exit gate.

## Building Goldens

- **From traces, when they exist:** run error
  analysis — open coding on ≥100 real traces (read
  them, tag failures in your own words, no fixed
  taxonomy yet), then axial coding to collapse those
  tags into 4–8 named failure buckets. Fewer than 4
  means the coding pass was too shallow; more than 8
  means buckets need merging. **Exception:**
  single-failure-surface tasks (e.g. strict-schema
  extraction) may land at 1–2 buckets with per-field
  sub-metrics inside one grader — don't invent
  artificial splits with no evidence behind them.
- **Synthetic, when traces don't exist yet:**
  dimension-based generation — enumerate the axes
  that matter (task type, difficulty, edge case,
  persona) and sample the cross-product; free-
  generated prompts cluster around whatever's
  easiest to write.
- **Goldens are versioned like code** — commit
  `eval/goldens.jsonl`, diff it in review, tag it per
  release. It doubles as the CI regression suite.

## Graders

One grader per failure bucket from error analysis —
not one for the whole eval set. A single blended
score hides which bucket regressed.

- **Deterministic first.** Regex, schema validation,
  or execution checks are cheaper, reproducible, and
  need no calibration.
- **LLM-judge only for genuinely subjective
  criteria** — tone, faithfulness, "which response
  is better" — where no deterministic check can
  express it.
- **Binary pass/fail over Likert.** A 1–5 or 1–10
  scale is noisier to calibrate and harder to apply
  consistently; collapse to pass/fail.
- **Drift-suite MMLU-style scoring: prefer logprob
  over generate-and-extract** — a tight token budget
  makes generate-and-extract parse-brittle for models
  that preamble, conflating format compliance with
  the knowledge being measured. Templates for all
  four grader shapes and this scoring note:
  `references/grader-templates.md`.

## Judge Calibration Is a Prerequisite

Any bucket routed to an LLM-judge needs calibration
before its verdicts count for anything beyond
exploration — a hard prerequisite, not a
nice-to-have. **N/A when no bucket routes to a
judge** — an all-deterministic harness has nothing
to calibrate; state that rather than leaving this
section unaddressed.

- Label ≥100 items, split **train**/**dev**/**sealed
  test** (report once, no re-touching after).
- Report **TPR and TNR**, not one blended accuracy
  number — a judge can hit 90% by always saying
  "pass" on a skewed set.
- **Pin the judge to a fixed model snapshot** and
  recalibrate on judge-model change, quarterly
  regardless.
- **The judge must come from a different model family
  than the model under test.**
- A judge that misses the agreed TPR/TNR bar ships
  **advisory-only** — flags for human review, never
  gates a promotion. Full protocol, bias correction,
  and recalibration checklist:
  `references/judge-calibration.md`.

## The Baseline

Before Phase 1 (method selection) starts, run the
full harness — goldens plus the capability-drift
suite — against the unmodified base model. This is
the number every later checkpoint gets compared
against.

`eval/baseline-<model>.json` is the gate token. No
baseline file, no comparison basis for
`checkpoint-promotion` — a checkpoint that "looks
better" against nothing measured isn't a finding.

## Directory Contract

```
eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint
```

`eval/` persists across runs and lives outside
`runs/` — the fixed measuring stick, not a run
artifact. `runs/` is disposable; `eval/` is not.
Never let a run script write into `eval/`. **Canonical
location:** every per-trace `results.json` — the
Phase 0 baseline included — lives at
`runs/<run-id>/results.json`, never under
`eval/runs/...`; an instruction requesting the
latter is wrong, not this contract.

### Phase 0 Exit Checklist

Before `finetuning-method-selection`, confirm:

1. ≥100 traces open-coded; 4–8 failure buckets (N/A
   floor for synthetic goldens on a single-failure-
   surface task — see the Building Goldens exception;
   bucket count then comes from post-baseline error
   analysis instead).
2. `eval/goldens.jsonl` committed and versioned.
3. One grader per bucket, deterministic first.
4. Judges calibrated — TPR/TNR, snapshot pinned,
   different family (**N/A when no bucket routes to
   an LLM-judge**; state that explicitly).
5. `eval/drift-suite.yaml` frozen.
6. `eval/baseline-<model>.json` written.

Missing any of the six (or its stated N/A)? Not
Phase 0 complete — `/finetune` checks the baseline
file before a run.

## Related Skills

General-purpose evaluation guidance (dashboards, A/B
testing, non-fine-tuning harnesses) lives in the
`llm-application-dev` plugin's `llm-evaluation`
skill — this skill covers only the fine-tuning
coupling: goldens that double as training data, and
the baseline that gates a checkpoint.

- `finetuning-method-selection` — routes here first.
- `dataset-curation` — formats these traces into
  training rows.
- `trace-to-training-data` — turns graded traces into
  training examples.
- `checkpoint-promotion` — consumes
  `baseline-<model>.json`, re-runs this harness on
  each candidate checkpoint.

## References

- `references/grader-templates.md` — runnable grader
  examples per shape, plus a `drift-suite.yaml`
  example and MMLU logprob-scoring note.
- `references/judge-calibration.md` — the
  calibration protocol, including the all-
  deterministic N/A path.

Run this skill on a real model without leaving the page. Every run is saved to your history for this skill.

How to use

1. 1Save the content below as SKILL.md in your agent's skills directory (e.g. .claude/skills/<name>/SKILL.md).
2. 2Or paste it directly into the conversation as context before asking the agent to do the task.
3. 3Adjust any project-specific paths or conventions mentioned in the skill to match your setup.