Batch experiments and evidence

A single fit covers one configuration. Answering whether a result holds at another seed, on another dataset or under another algorithm requires many runs. kd.harness runs a matrix of configurations and records each one, so the aggregate is computed from the records rather than assembled by hand.

The experiment plan

An ExperimentPlan is an ordered matrix of entries, each naming an algorithm, a dataset, a seed and the settings for that run. Its hash covers the matrix including the order, so the hash identifies the exact study. Datasets are referred to by name and resolved at run time, which keeps the plan itself free of data.

import kd
from kd.harness import ExperimentPlan, PlanEntry, run_plan

dataset = kd.generate_burgers_data(nx=64, nt=32, nu=0.1, seed=0)

plan = ExperimentPlan(
    name="burgers-seed-stability",
    entries=tuple(
        PlanEntry(
            instrument="sga",
            dataset_ref="burgers",
            seed=seed,
            model_kwargs={"generations": 6, "population": 12},
        )
        for seed in (0, 1, 2)
    ),
)
print(plan.name, len(plan.entries), plan.plan_hash()[:10])

result = run_plan(plan, datasets={"burgers": dataset}, store_root="out/burgers-seeds")
for outcome in result.outcomes:
    print(outcome.entry_index, outcome.entry.instrument, outcome.entry.seed, outcome.status)

burgers-seed-stability 3 1ec66c4dac
0 sga 0 completed
1 sga 1 completed
2 sga 2 completed

Widening the study means editing those entries to add an algorithm, a dataset or more seeds. The surrounding code does not change, and the new matrix receives its own hash.

run_plan runs the entries one after another in the calling process, so the batch ends if that process dies, and the outcomes written before that point remain in the store. To run a study in parallel, split its entries in order into several plans with the same name, run each plan in its own process with its own store directory, and pass the stores to merge_stores in the order of the split, under that name. Batches that ran separately for another reason, for example on different days or machines, are combined the same way.

merge_stores writes a new store from several sealed ones and leaves the sources unchanged. The merged plan is the sources' entries concatenated in the order given, so a single run_plan over the same entries under the same name has the same plan hash and the same entry indices. Records are copied unchanged, and the new store is reopened and re-verified after the copy. The merge is refused when one entry was attempted in two sources, when one dataset name refers to different data in two sources, or when a source is itself a merged store. An entry counts as attempted once it has an outcome of any status, so an entry that failed in one batch and was run again in another cannot be merged back with the first batch. An entry that a batch never reached can be run in a later batch and merged.

The merged store keeps one row per source with its plan name, plan hash, entry range and environment, and its top-level environment holds only the fields on which all sources agree. When the sources ran under different KD versions or commits, the merge requires a version_note stating what changed between them and whether it affects the results. The store report then lists each source's KD version and commit together with the note. The consensus report lists them as well when it is rendered with the merged store, render_consensus_markdown(report, store=merged).

Failed entries

An entry that raises is recorded with its error type and message, and the remaining entries still run. Each outcome carries one of three statuses: the fit completed, it raised, or it finished without sealing a record. A failure to write the evidence stops the batch, because the outcomes of the remaining entries could not be recorded.

The evidence store

Records are written atomically and never overwritten, and the index is rewritten after every outcome, so an interrupted batch keeps everything that finished. The directory must be fresh, since a store is never appended to.

Reopening a store re-verifies it. The plan hash is recomputed from the persisted plan, every record is verified against its own hash and against the plan slot it claims, records under one dataset name must share one data fingerprint, and a file in the records directory that the index does not list raises an error. The reopened store is read-only. Each store also records the environment it ran in: the KD, Python and torch versions, the platform string, and the commit if the run happened inside a checkout.

Consensus across runs

build_consensus is a pure function over a sealed store. Runs are grouped by the canonical set of terms they selected, so two runs that print different strings but selected the same law are placed in the same class:

from kd.harness import EvidenceStore, build_consensus

store = EvidenceStore.load("out/burgers-seeds")
report = build_consensus(store)
for dataset_consensus in report.datasets:
    for cls in dataset_consensus.classes:
        members = ", ".join(f"{m.instrument}/seed{m.seed}" for m in cls.members)
        print(cls.structure_key[:10], "|", members)
        print("   ", cls.terms)
    for pair in dataset_consensus.adjacency:
        print("adjacency", pair.structure_key_a[:10], pair.structure_key_b[:10],
              pair.relation, round(pair.jaccard, 2))

ed0de3fa1f | sga/seed0, sga/seed1
    ('diff_x(u_x)', 'mul(u,u_x)', 'u_t')
2b7f162ff3 | sga/seed2
    ('diff2_x(sub(u,x))', 'diff_x(n2(u))', 'u_t')
adjacency ed0de3fa1f 2b7f162ff3 disjoint 0.0

The three runs produced two classes. The second class is the same law in a different notation: diff_x(n2(u)) is the derivative of u², which equals 2·u·u_x, and diff2_x(sub(u, x)) is the second derivative of u - x, which equals u_xx. Seed 2 therefore selected the same law written differently. Term sets are grouped rather than simplified, and the adjacency line compares them literally, leaving out the shared left-hand side u_t, so it reports the two classes as disjoint: it does not detect that two spellings evaluate to the same column. To check whether a split is a difference of notation, fit both term sets on the data with kd.evaluate_terms and compare the recovered laws.

Each class is described along four independent axes: whether the structure agrees, whether the support agrees, whether the coefficients agree, and whether the empirical fit agrees. The four are reported separately, without a combined confidence number, because they can disagree with one another. The report also records how far the corroboration extends, from several algorithms across different derivative sources down to a single run, so a class supported by three seeds of one algorithm is not read as a class supported by three algorithms.

The report renders to Markdown and to a JSON artifact. Both are deterministic: the same evidence renders byte for byte the same way.