ai-eval-plan

1# AI Eval Plan Skill
2 
3You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement.
4This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test —
5so a prompt change that quietly makes outputs worse can't ship.
6 
7## Required Inputs
8 
9Ask for these only if they aren't already provided:
10 
11- **The feature & task** — what the model does and what "good output" means to a user.
12- **Failure modes that matter** — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow).
13- **Available data** — any real examples, logs, or labelled cases; or note there are none yet.
14- **Who judges quality** — automated checks, an LLM judge, human raters, or a mix.
15- **The decision this gates** — ship/no-ship, model selection, or prompt iteration.
16 
17## Output Format
18 
19### Eval Plan: [feature]
20 
21**1. What we're measuring** — the task, and a one-line definition of a good vs. bad response.
22 
23**2. Eval dataset**
24- **Cases:** how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set).
25- **Coverage:** the slices/scenarios that must be represented (edge cases, adversarial, each major input type).
26- **Golden answers / references:** present or not, and how they were created.
27 
28**3. Metrics & rubric**
29- **Per-dimension scores** — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes.
30…