second-brain-backfill

1# Backfill an archive
2 
3A backfill is where cost and quality both go wrong quietly. Done in the wrong
4order it also inverts the vault's history, because later sources should update
5pages that earlier ones created.
6 
7## Core rule
8 
9Oldest first, ten sources per batch, stop for a go-ahead between batches.
10 
11## Workflow
12 
131. **Triage before ingesting.** Read titles and first paragraphs. Sort into
14   worth-ingesting, keep-in-raw, and delete. Most archives are half dead links
15   and things saved but never opened.
162. **Order by date, oldest first.** Run it the other way and every old source
17   arrives as a contradiction against a page that already has the final answer.
183. **Batch of ten.** Ingest following `second-brain-ingest`.
194. **Checkpoint:** update `index.md`, append to `log.md`, commit with the batch
20   range in the message, report counts.
215. **Stop and wait** before the next batch.
226. **Full lint at the end.** Large imports always leave orphans and duplicates.
23 
24## Output format
25 
26```
27Batch <n>/<total>: <range>
28Ingested: <n> sources -> <n> new pages, <n> updated, <n> links
29Skipped: <n> (reason)
30Running total: <n> of <n>
31Next batch: <what it contains>
32```
33 
34## Calibration
35 
36Read every page the first batch produced, all of them. A flaw you let through in
37batch one is a flaw in four hundred pages by the end, and fixing it afterwards
38costs more than the original run.
39 
40If the archive is over about two hundred sources, propose a filter before
41starting rather than quoting the full cost.