advanced-evaluation

1# Advanced Evaluation
2 
3This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
4 
5**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
6 
7## When to Use
8Activate this skill when:
9 
10- Building automated evaluation pipelines for LLM outputs
11- Comparing multiple model responses to select the best one
12- Establishing consistent quality standards across evaluation teams
13- Debugging evaluation systems that show inconsistent results
14- Designing A/B tests for prompt or model changes
15- Creating rubrics for human or automated evaluation
16- Analyzing correlation between automated and human judgments
17 
18## Core Concepts
19 
20### The Evaluation Taxonomy
21 
22Evaluation approaches fall into two primary categories with distinct reliability profiles:
23 
24**Direct Scoring**: A single LLM rates one response on a defined scale.
25- Best for: Objective criteria (factual accuracy, instruction following, toxicity)
26- Reliability: Moderate to high for well-defined criteria
27- Failure mode: Score calibration drift, inconsistent scale interpretation
28 
29**Pairwise Comparison**: An LLM compares two responses and selects the better one.
30…