02-ref-hallucination-arena

1# Reference Hallucination Arena Skill
2 
3Evaluate how accurately LLMs recommend real academic references using the
4OpenJudge `RefArenaPipeline`:
5 
61. **Load queries** — from JSON/JSONL dataset
72. **Collect responses** — BibTeX-formatted references from target models
83. **Extract references** — parse BibTeX entries from model output
94. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP
105. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown
116. **Generate report** — Markdown report + visualization charts
12 
13## Prerequisites
14 
15```bash
16# Install OpenJudge
17pip install py-openjudge
18 
19# Extra dependency for ref_hallucination_arena (chart generation)
20pip install matplotlib
21```
22 
23## Gather from user before running
24 
25| Info | Required? | Notes |
26|------|-----------|-------|
27| Config YAML path | Yes | Defines endpoints, dataset, verification settings |
28| Dataset path | Yes | JSON/JSONL file with queries (can be set in config) |
29| API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. |
30| CrossRef email | No | Improves API rate limits for verification |
31| PubMed API key | No | Improves PubMed rate limits |
32| Output directory | No | Default: `./evaluation_results/ref_hallucination_arena` |
33| Report language | No | `"en"` (default) or `"zh"` |
34| Tavily API key | No | Required only if using tool-augmented mode |
35 
36## Quick start
37 
38### CLI
39 
40```bash
41# Run evaluation with config file
42…