Batch experiments and evidence

A single fit covers one configuration. Answering whether a result holds at another seed, on another dataset or under another algorithm requires many runs. kd.harness runs a matrix of configurations and records each one, so the aggregate is computed from the records rather than assembled by hand.

The experiment plan

An ExperimentPlan is an ordered matrix of entries, each naming an algorithm, a dataset, a seed and the settings for that run. Its hash covers the matrix including the order, so the hash identifies the exact study. Datasets are referred to by name and resolved at run time, which keeps the plan itself free of data.

import kd
from kd.harness import ExperimentPlan, PlanEntry, run_plan

dataset = kd.generate_burgers_data(nx=64, nt=32, nu=0.1, seed=0)

plan = ExperimentPlan(
    name="burgers-seed-stability",
    entries=tuple(
        PlanEntry(
            instrument="sga",
            dataset_ref="burgers",
            seed=seed,
            model_kwargs={"generations": 6, "population": 12},
        )
        for seed in (0, 1, 2)
    ),
)
print(plan.name, len(plan.entries), plan.plan_hash()[:10])

result = run_plan(plan, datasets={"burgers": dataset}, store_root="out/burgers-seeds")
for outcome in result.outcomes:
    print(outcome.entry_index, outcome.entry.instrument, outcome.entry.seed, outcome.status)

burgers-seed-stability 3 1ec66c4dac
0 sga 0 completed
1 sga 1 completed
2 sga 2 completed

Widening the study means editing those entries to add an algorithm, a dataset or more seeds. The surrounding code does not change, and the new matrix receives its own hash.

Entries run sequentially. Parallel execution is a separate call that shards a plan across subprocesses. It pins the numerical libraries to one thread per worker, limits the memory-heavy algorithms to one at a time, and gives each shard a wall-clock limit. The shards are then merged back into a single store, which replays every record and checks that none of them changed during the merge.

Failed entries

An entry that raises is recorded with its error type and message, and the remaining entries still run. Each outcome carries one of three statuses: the fit completed, it raised, or it finished without sealing a record. A failure to write the evidence stops the batch, because the outcomes of the remaining entries could not be recorded.

The evidence store

Records are written atomically and never overwritten, and the index is rewritten after every outcome, so an interrupted batch keeps everything that finished. The directory must be fresh, since a store is never appended to.

Reopening a store re-verifies it. The plan hash is recomputed from the persisted plan, every record is verified against its own hash and against the plan slot it claims, records under one dataset name must share one data fingerprint, and a file in the records directory that the index does not list raises an error. The reopened store is read-only. Each store also records the environment it ran in: the KD, Python and torch versions, the platform string, and the commit if the run happened inside a checkout.

Consensus across runs

build_consensus is a pure function over a sealed store. Runs are grouped by the canonical set of terms they selected, so two runs that print different strings but selected the same law are placed in the same class:

from kd.harness import EvidenceStore, build_consensus

store = EvidenceStore.load("out/burgers-seeds")
report = build_consensus(store)
for dataset_consensus in report.datasets:
    for cls in dataset_consensus.classes:
        members = ", ".join(f"{m.instrument}/seed{m.seed}" for m in cls.members)
        print(cls.structure_key[:10], "|", members)
        print("   ", cls.terms)
    for pair in dataset_consensus.adjacency:
        print("adjacency", pair.structure_key_a[:10], pair.structure_key_b[:10],
              pair.relation, round(pair.jaccard, 2))

9f6aa9b928 | sga/seed0, sga/seed1
    ('diff_x(u_x)', 'mul(u,u_x)', 'u_t')
0bb825b5f3 | sga/seed2
    ('diff2_x(sub(u,x))', 'diff_x(n2(u))', 'u_t')
adjacency 9f6aa9b928 0bb825b5f3 overlap 0.2

The three runs produced two classes. The second class is the same law in a different notation: diff_x(n2(u)) is the derivative of u², which equals 2·u·u_x, and diff2_x(sub(u, x)) is the second derivative of u - x, which equals u_xx. Seed 2 therefore selected the same law written differently, and the adjacency line reports the overlap between the two classes. Term sets are grouped rather than simplified, so the report records a relation between classes instead of merging them.

Each class is described along four independent axes: whether the structure agrees, whether the support agrees, whether the coefficients agree, and whether the empirical fit agrees. The four are reported separately, without a combined confidence number, because they can disagree with one another. The report also records how far the corroboration extends, from several algorithms across different derivative sources down to a single run, so a class supported by three seeds of one algorithm is not read as a class supported by three algorithms.

The report renders to Markdown and to a JSON artifact. Both are deterministic: the same evidence renders byte for byte the same way.