datalad

1# DataLad
2 
3## Overview
4 
5DataLad is a data management layer over Git and git-annex. Git tracks the dataset
6structure, small text files, and the history. git-annex tracks the *content* of large
7files, storing each file as a key and keeping the bytes somewhere that is not necessarily
8the local repository.
9 
10A normal clone retrieves Git history and the top-level file listing while leaving
11annexed bytes unfetched. Installed subdatasets have their own histories; a clone does not
12automatically populate them. Clone cost depends on Git history and file count, not just
13the data volume. Retrieve annexed bytes selectively with `datalad get`.
14 
15The second thing DataLad adds is provenance. `datalad run` executes a command and commits
16the result together with a machine-readable record of the command, its inputs, and its
17outputs. `datalad rerun` reads that record back and re-executes it. This turns "how was
18this figure produced" from an archaeology problem into a command.
19 
20## When to use DataLad instead of plain Git
21 
22Use DataLad when any of the following holds:
23 
24- Files are too large for Git to handle comfortably, or the total exceeds what every
25  collaborator wants on disk.
26- Data lives in more than one place (a lab server, a cluster scratch, S3, a supercomputer)
27  and you need to know which copies exist.
28- The analysis must be re-executable, and a plain commit message is not enough evidence.
29- You are consuming published datasets from OpenNeuro, DANDI, or `datasets.datalad.org`,
30…