Skills for Real Engineers. Straight from my .agents directory.
39 skills
01-graders-and-pipeline
Agent skill from
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.
How to install 01-graders-and-pipeline
install 01-graders-and-pipeline
# install the OpenJudge pack with the skills CLI
npx skills add agentscope-ai/OpenJudge# or add only this skill to Claude Code: copy the folder
skills/openjudge-core/01-graders-and-pipeline/
# into
~/.claude/skills/01-graders-and-pipeline/ (all projects)
.claude/skills/01-graders-and-pipeline/ (one project)
1# OpenJudge Skill23Build evaluation pipelines for LLM applications using the `openjudge` library.45## When to Use This Skill67- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)8- User wants to compare two or more models and rank them9- User wants to design a scoring rubric and automate evaluation10- User wants to analyze evaluation results statistically11- User wants to build a reward model or quality filter1213## Sub-documents — Read When Relevant1415| Topic | File | Read when… |16|-------|------|------------|17| Grader selection & configuration | `graders.md` | User needs to pick or configure an evaluator |18| Batch evaluation pipeline | `pipeline.md` | User needs to run evaluation over a dataset |19| Auto-generate graders from data | `generator.md` | No rubric yet; generate from labeled examples |20| Analyze & compare results | `analyzer.md` | User wants win rates, statistics, or metrics |2122Read the relevant sub-document **before** writing any code.2324## Install2526```bash27pip install py-openjudge28```2930## Architecture Overview3132```33Dataset (List[dict])34 │35 ▼36GradingRunner ← orchestrates everything37 │38 ├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank39 ├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank40 └─► Grader C ...41 │42 ├─► Aggregator (optional) ← combine multiple grader scores into one43 │44…
Excerpt shown under the repository's Apache-2.0 license.
More skills in OpenJudge
Keep exploring
Similar skill repositories
An agentic skills framework & software development methodology that works.
15 skills
A single CLAUDE.md file to improve Claude Code behavior, derived from Andrej Karpathy's observations on LLM coding pitfalls.
CC1 skill
Public repository for Agent Skills
20 skills
An AI skill that provides design intelligence for building professional UI/UX across multiple platforms.
CCCXCUGH6 skills
Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
CCCXGHGM+135 skills