ml-training-recipes

1# ML Training Recipes
2
3Battle-tested patterns for PyTorch training across domains. Drawn from production codebases
4(Karpathy's autoresearch/nanochat, torchvision, HuggingFace) and modern training practice.
5
6## Reference files (read when needed)
7
8- `references/architecture.md` — Transformer/LLM architecture code patterns, weight init
9- `references/optimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps
10- `references/domain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading
11- `references/scaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark
12- `references/biomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP
13- `references/experiment-loop.md` — Autonomous experiment loop (autoresearch keep/discard/revert)
14
15---
16
17## Architecture Selection
18
19Pick the right model by **data type** and **data scale**:
20
21| Data Type | < 10K samples | 10K-100K | > 100K |
22|-----------|--------------|----------|--------|
23| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |
24| **Text (gen)** | Few-shot prompting | Fine-tune GPT/LLaMA (LoRA) | Pretrain from scratch |
25| **Tabular** | XGBoost/LightGBM | Still XGBoost | Neural viable |
26| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |
27| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |
28…