evaluating-llms-harness

1# lm-evaluation-harness - LLM Benchmarking
2 
3## Quick start
4 
5lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.
6 
7**Installation**:
8```bash
9pip install lm-eval
10```
11 
12**Evaluate any HuggingFace model**:
13```bash
14lm_eval --model hf \
15  --model_args pretrained=meta-llama/Llama-2-7b-hf \
16  --tasks mmlu,gsm8k,hellaswag \
17  --device cuda:0 \
18  --batch_size 8
19```
20 
21**View available tasks**:
22```bash
23lm_eval --tasks list
24```
25 
26## Common workflows
27 
28### Workflow 1: Standard benchmark evaluation
29 
30Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).
31 
32Copy this checklist:
33 
34```
35Benchmark Evaluation:
36- [ ] Step 1: Choose benchmark suite
37- [ ] Step 2: Configure model
38- [ ] Step 3: Run evaluation
39- [ ] Step 4: Analyze results
40```
41 
42**Step 1: Choose benchmark suite**
43 
44**Core reasoning benchmarks**:
45- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice
46- **GSM8K** - Grade school math word problems
47- **HellaSwag** - Common sense reasoning
48- **TruthfulQA** - Truthfulness and factuality
49- **ARC** (AI2 Reasoning Challenge) - Science questions
50 
51**Code benchmarks**:
52- **HumanEval** - Python code generation (164 problems)
53- **MBPP** (Mostly Basic Python Problems) - Python coding
54 
55**Standard suite** (recommended for model releases):
56```bash
57--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge
58```
59 
60**Step 2: Configure model**
61 
62**HuggingFace model**:
63```bash
64lm_eval --model hf \
65…