ai-pipeline-orchestration

1# AI Pipeline Orchestration
2 
3Build reliable, observable AI workflows — from document ingestion to batch inference to model training pipelines.
4 
5## When to Use This Skill
6 
7Use this skill when:
8- Scheduling recurring RAG document ingestion and re-indexing
9- Orchestrating multi-step batch LLM processing workflows
10- Running nightly model evaluation and fine-tuning jobs
11- Building ETL pipelines that feed into AI models
12- Managing dependencies between data preparation and model serving
13 
14## Tool Selection
15 
16| Tool | Best For | Complexity | GPU Jobs |
17|------|----------|------------|----------|
18| **Prefect** | Modern Python-first; easy to adopt | Low | Good |
19| **Airflow** | Complex DAGs; large teams; existing usage | High | Good |
20| **Dagster** | Asset-centric; strong data lineage | Medium | Excellent |
21| **Temporal** | Long-running workflows; reliability-first | Medium | Good |
22 
23## Prefect — Quick Start
24 
25```bash
26pip install prefect prefect-kubernetes
27 
28# Start Prefect server (or use Prefect Cloud)
29prefect server start
30 
31# In another terminal
32prefect worker start --pool default-agent-pool
33```
34 
35## Prefect: RAG Ingestion Pipeline
36 
37```python
38from prefect import flow, task, get_run_logger
39from prefect.tasks import task_input_hash
40from datetime import timedelta
41import hashlib
42 
43@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=24))
44def fetch_documents(source_url: str) -> list[dict]:
45    """Fetch documents from source; cached to avoid re-fetching."""
46…