01-auto-arena

1# Auto Arena Skill
2 
3End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`:
4 
51. **Generate queries** — LLM creates diverse test queries from task description
62. **Collect responses** — query all target endpoints concurrently
73. **Generate rubrics** — LLM produces evaluation criteria from task + sample queries
84. **Pairwise evaluation** — judge model compares every model pair (with position-bias swap)
95. **Analyze & rank** — compute win rates, win matrix, and rankings
106. **Report & charts** — Markdown report + win-rate bar chart + optional matrix heatmap
11 
12## Prerequisites
13 
14```bash
15# Install OpenJudge
16pip install py-openjudge
17 
18# Extra dependency for auto_arena (chart generation)
19pip install matplotlib
20```
21 
22## Gather from user before running
23 
24| Info | Required? | Notes |
25|------|-----------|-------|
26| Task description | Yes | What the models/agents should do (set in config YAML) |
27| Target endpoints | Yes | At least 2 OpenAI-compatible endpoints to compare |
28| Judge endpoint | Yes | Strong model for pairwise evaluation (e.g. `gpt-4`, `qwen-max`) |
29| API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. |
30| Number of queries | No | Default: `20` |
31| Seed queries | No | Example queries to guide generation style |
32| System prompts | No | Per-endpoint system prompts |
33| Output directory | No | Default: `./evaluation_results` |
34| Report language | No | `"zh"` (default) or `"en"` |
35 
36## Quick start
37 
38### CLI
39 
40```bash
41# Run evaluation
42…