aaai-experiments

1# AAAI Experiments
2 
3Use this before submission to ensure empirical evidence supports the AI contribution. AAAI
4reviewers may come from adjacent AI subfields, so experiments must be interpretable beyond one
5benchmark community.
6 
7## Experiment audit
8 
9- Map every experimental block to a claim in the introduction.
10- Compare against strong, recent, and fairly tuned baselines.
11- Include ablations that isolate mechanisms rather than removing multiple components at once.
12- Report uncertainty, variance, and statistical tests when small differences matter.
13- Test robustness to data split, prompt, seed, environment, user population, or distribution shift
14  when relevant.
15- For human evaluation, document task, instructions, annotator pool, quality control, aggregation,
16  and ethics/IRB status.
17- Report compute, hardware, data access, model size, and training/inference cost.
18 
19## Claim-to-evidence ledger
20 
21Build this table before adding new experiments. It keeps the AAAI evidence package aligned with
22the main text and with the reproducibility checklist.
23 
24| Manuscript claim | Required evidence | Phase-1 risk if missing | Checklist hook |
25| --- | --- | --- | --- |
26| New AI capability | benchmark + qualitative failure cases | broad reviewer sees only engineering | datasets, metrics, baselines |
27| Better mechanism | single-factor ablations | gain looks like tuning luck | ablation and hyperparameter answers |
28…