generate-synthetic-data

1# Generate Synthetic Data
2 
3Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.
4 
5## Prerequisites
6 
7Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.
8 
9## Core Process
10 
11### Step 1: Define Dimensions
12 
13Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.
14 
15```
16Dimension 1: [Name] — [What it captures]
17  Values: [value_a, value_b, value_c, ...]
18 
19Dimension 2: [Name] — [What it captures]
20  Values: [value_a, value_b, value_c, ...]
21 
22Dimension 3: [Name] — [What it captures]
23  Values: [value_a, value_b, value_c, ...]
24```
25 
26Example for a real estate assistant:
27 
28```
29Feature: what task the user wants
30  Values: [property search, scheduling, email drafting]
31 
32Client Persona: who the user serves
33  Values: [first-time buyer, investor, luxury buyer]
34 
35Scenario Type: query clarity
36  Values: [well-specified, ambiguous, out-of-scope]
37```
38 
39Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.
40 
41### Step 2: Draft 20 Tuples with the User
42…