data-quality-frameworks

1# Data Quality Frameworks
2 
3Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.
4 
5## When to Use This Skill
6 
7- Implementing data quality checks in pipelines
8- Setting up Great Expectations validation
9- Building comprehensive dbt test suites
10- Establishing data contracts between teams
11- Monitoring data quality metrics
12- Automating data validation in CI/CD
13 
14## Core Concepts
15 
16### 1. Data Quality Dimensions
17 
18| Dimension | Description | Example Check |
19|-----------|-------------|---------------|
20| **Completeness** | No missing values | `expect_column_values_to_not_be_null` |
21| **Uniqueness** | No duplicates | `expect_column_values_to_be_unique` |
22| **Validity** | Values in expected range | `expect_column_values_to_be_in_set` |
23| **Accuracy** | Data matches reality | Cross-reference validation |
24| **Consistency** | No contradictions | `expect_column_pair_values_A_to_be_greater_than_B` |
25| **Timeliness** | Data is recent | `expect_column_max_to_be_between` |
26 
27### 2. Testing Pyramid for Data
28 
29```
30          /\
31         /  \     Integration Tests (cross-table)
32        /────\
33       /      \   Unit Tests (single column)
34      /────────\
35     /          \ Schema Tests (structure)
36    /────────────\
37```
38 
39## Quick Start
40 
41### Great Expectations Setup
42 
43```bash
44# Install
45pip install great_expectations
46 
47# Initialize project
48great_expectations init
49 
50# Create datasource
51great_expectations datasource new
52```
53…