aamas-experiments

1# AAMAS Experiments
2 
3Use this before submission when the empirical or simulation story is not yet locked. At AAMAS
4the experiment exists to test the *interaction* claim, not to top a benchmark.
5 
6## Experiment audit
7 
8- Map each empirical claim to a game, a self-play run, a population sweep, an ablation, or a
9  deviation test.
10- Choose opponents deliberately: self-play alone rarely suffices; include held-out opponents,
11  population sets, or classical strategies as the claim requires.
12- Separate simulations that validate a solution concept (where the equilibrium is known) from
13  real or applied studies that show practical multiagent behavior.
14- Report uncertainty for stochastic results over both seeds and opponents: standard errors,
15  confidence intervals, or paired tests.
16- Report the environment, number of agents, training regime, evaluation protocol, metrics,
17  hyperparameter ranges, chosen settings, seeds, hardware, software versions, and runtime.
18- Add ablations for the interaction mechanism (communication, reward sharing, the payment rule),
19  not just cosmetic variants.
20- Audit for the mismatch between the strategic claim and the setup: an equilibrium claim tested
21  against only one fixed opponent, or a cooperation claim that hides a reward-shaping constant.
22 
23## What experiments are for at this venue
24 
25- The strongest design shows the interaction under stress: agents that *can* deviate, opponents
26…