ml-pipeline

1# ML Pipeline Expert
2 
3Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.
4 
5## Core Workflow
6 
71. **Design pipeline architecture** — Map data flow, identify stages, define interfaces between components
82. **Validate data schema** — Run schema checks and distribution validation before any training begins; halt and report on failures
93. **Implement feature engineering** — Build transformation pipelines, feature stores, and validation checks
104. **Orchestrate training** — Configure distributed training, hyperparameter tuning, and resource allocation
115. **Track experiments** — Log metrics, parameters, and artifacts; enable comparison and reproducibility
126. **Validate and deploy** — Run model evaluation gates; implement A/B testing or shadow deployment before promotion
13 
14## Reference Guide
15 
16Load detailed guidance based on context:
17 
18| Topic | Reference | Load When |
19|-------|-----------|-----------|
20| Feature Engineering | `references/feature-engineering.md` | Feature pipelines, transformations, feature stores, Feast, data validation |
21| Training Pipelines | `references/training-pipelines.md` | Training orchestration, distributed training, hyperparameter tuning, resource management |
22| Experiment Tracking | `references/experiment-tracking.md` | MLflow, Weights & Biases, experiment logging, model registry |
23…