llm-eval-harness

1# LLM Eval Harness
2 
3## Overview
4 
5Give this skill an endpoint (`base_url` + `model` + an API key in an env var) and it
6measures whether the endpoint actually works and whether the model is fast, stable,
7protocol-correct, and good enough — instead of trusting the vendor's headline numbers.
8Choose from the following evaluation dimensions:
9 
10| Dimension | Script | Answers |
11|---|---|---|
12| **Availability** | `scripts/availability_probe.py` | which model IDs work here, with explicit error classification |
13| **Request fidelity** | `scripts/fidelity_probe.py` | do system prompt / tools / history actually REACH the model? |
14| **Speed** | `scripts/speed_probe.py` | TTFT + sustained decode tok/s, **thinking-aware** |
15| **Concurrency / stability** | `scripts/concurrency_probe.py` | success rate, p50/p90 latency, where it breaks |
16| **Protocol compliance** | `scripts/protocol_probe.py` | does the Anthropic `thinking` block actually fire when requested, AND does the endpoint accept one already sitting in history on a later turn (N≥10, both are separate checks — see Dimension 3)? |
17| **Quality / use-case regression** | `scripts/usecase_runner.py` + blind judges | does it pass *your* accumulated cases? |
18 
19## Which dimensions to run
20 
21Route from what the user is actually doing:
22 
23- "接入新模型/新网关先测一下" / onboarding a provider → **Availability → Fidelity**,
24  then speed/concurrency if adoption looks likely
25- "这个 ID 能不能用 / 写个支持列表" → **Availability** (and read disciplines §9 before
26…