awq-quantization

1# AWQ (Activation-aware Weight Quantization)
2 
34-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.
4 
5## When to use AWQ
6 
7**Use AWQ when:**
8- Need 4-bit quantization with <5% accuracy loss
9- Deploying instruction-tuned or chat models (AWQ generalizes better)
10- Want ~2.5-3x inference speedup over FP16
11- Using vLLM for production serving
12- Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support
13 
14**Use GPTQ instead when:**
15- Need maximum ecosystem compatibility (more tools support GPTQ)
16- Working with ExLlamaV2 backend specifically
17- Have older GPUs without Marlin support
18 
19**Use bitsandbytes instead when:**
20- Need zero calibration overhead (quantize on-the-fly)
21- Want to fine-tune with QLoRA
22- Prefer simpler integration
23 
24## Quick start
25 
26### Installation
27 
28```bash
29# Default (Triton kernels)
30pip install autoawq
31 
32# With optimized CUDA kernels + Flash Attention
33pip install autoawq[kernels]
34 
35# Intel CPU/XPU optimization
36pip install autoawq[cpu]
37```
38 
39**Requirements**: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+
40 
41### Load pre-quantized model
42 
43```python
44from awq import AutoAWQForCausalLM
45from transformers import AutoTokenizer
46 
47model_name = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"
48 
49model = AutoAWQForCausalLM.from_quantized(
50    model_name,
51    fuse_layers=True  # Enable fused attention for speed
52)
53tokenizer = AutoTokenizer.from_pretrained(model_name)
54 
55# Generate
56…