modal-serverless-gpu

1# Modal Serverless GPU
2 
3Comprehensive guide to running ML workloads on Modal's serverless GPU cloud platform.
4 
5## When to use Modal
6 
7**Use Modal when:**
8- Running GPU-intensive ML workloads without managing infrastructure
9- Deploying ML models as auto-scaling APIs
10- Running batch processing jobs (training, inference, data processing)
11- Need pay-per-second GPU pricing without idle costs
12- Prototyping ML applications quickly
13- Running scheduled jobs (cron-like workloads)
14 
15**Key features:**
16- **Serverless GPUs**: T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand
17- **Python-native**: Define infrastructure in Python code, no YAML
18- **Auto-scaling**: Scale to zero, scale to 100+ GPUs instantly
19- **Sub-second cold starts**: Rust-based infrastructure for fast container launches
20- **Container caching**: Image layers cached for rapid iteration
21- **Web endpoints**: Deploy functions as REST APIs with zero-downtime updates
22 
23**Use alternatives instead:**
24- **RunPod**: For longer-running pods with persistent state
25- **Lambda Labs**: For reserved GPU instances
26- **SkyPilot**: For multi-cloud orchestration and cost optimization
27- **Kubernetes**: For complex multi-service architectures
28 
29## Quick start
30 
31### Installation
32 
33```bash
34pip install modal
35modal setup  # Opens browser for authentication
36```
37 
38### Hello World with GPU
39 
40```python
41import modal
42 
43app = modal.App("hello-gpu")
44 
45@app.function(gpu="T4")
46def gpu_info():
47    import subprocess
48…