Parameter tiers and resuming

Whether a run can be resumed depends on which parameters changed between the checkpoint and the new kd.Model.

The parameter table on every algorithm page has a "Resume" column, and every parameter carries a badge there: resume-safe or init-only. The algorithm declares that badge itself, and the badge determines whether a new value takes effect on a resume.

Tier	Meaning
resume-safe	the new value takes effect when resuming from a checkpoint, and the stored search state stays valid
init-only	the new value applies to a fresh fit only; resuming with the change raises

A third tier is not shown in the tables. Fields that change the scientific identity of the run, such as the candidate library and derivative semantics, are identity-breaking: changing one makes the two segments a different experiment, and the resume raises.

Any config field with no declared tier is treated as init-only. The default refuses because the two mistakes have different consequences. A field that should have been declared resume-safe requires one line of edit to correct, while a resume that succeeded across two different configurations leaves the error in the result.

The tables use the config field names. Two fields are spelled differently as kd.Model keywords: num is population, and use_autograd is derivatives.

Checkpoint writing

Setting checkpoint_dir to a directory enables checkpointing. checkpoint_every sets the interval in iterations, and a checkpoint_final.pt is written when the run ends. The directory is managed by a manifest.json and cannot be reused: pointing at a non-empty directory raises, so each run needs a fresh directory. checkpoint_keep_last bounds how many periodic checkpoints are retained, and the final one is neither counted nor pruned.

import kd

dataset = kd.load_kdv()
model = kd.Model(
    algorithm="sga", generations=4, seed=42, verbose=False,
    checkpoint_dir="out/kdv-ckpt", checkpoint_every=2,
)
model.fit(dataset)

The manifest

kd.load_checkpoint_manifest returns the directory's entries in write order. It is read-only and performs no repair or cleanup. Select a checkpoint from these entries rather than globbing the directory.

for entry in kd.load_checkpoint_manifest("out/kdv-ckpt"):
    print(entry.filename, entry.kind, entry.final_status, entry.iteration, entry.best_score)

checkpoint_000000.pt periodic None 0 -18.37657775848694
checkpoint_000002.pt periodic None 2 -18.37657775848694
checkpoint_final.pt final completed 3 -18.37657775848694

An entry also carries best_expression, algorithm, seed, created_at, kd_version and more. kind separates periodic from final checkpoints, and final_status is set on the final one only: completed for a run that finished, crashed for the final checkpoint a crashed run left behind, so the two are distinguishable. Both values have exported constants, kd.KIND_FINAL and kd.FINAL_STATUS_COMPLETED.

Resume a run

fit takes a checkpoint path as resume_from. A resume restores the search state: the population, the controller weights, and the current best. The number of generations comes from this kd.Model, and the parameters are judged by the tiers above. seed is not in the tier table, so it takes the unregistered-field default of init-only. A plain resume keeps the seed the checkpoint was written with, and a different seed is rejected. Passing reseed=True to fit turns the resume into a branch: it keeps the restored search state and re-derives the random streams from this model's seed, so a changed seed is accepted. Branching is available for the algorithms whose segmentation.reseed declaration is true, today sga, dlga and discover.

resumed = kd.Model(algorithm="sga", generations=8, seed=42, verbose=False, population=25)
resumed.fit(dataset, resume_from="out/kdv-ckpt/checkpoint_final.pt")
print(resumed.best_expr_)

u_t = -0.0025*diff2_x(diff_x(u)) - 1*mul(u_x, u)

population is resume-safe, so the resume accepts the new value. With the init-only depth instead:

changed = kd.Model(algorithm="sga", generations=8, seed=42, verbose=False, depth=5)
changed.fit(dataset, resume_from="out/kdv-ckpt/checkpoint_final.pt")

ValueError: resume config mismatch for algorithm 'sga': init_only field(s) changed
vs the checkpoint: depth (stored 4 -> live 5). These fields can only be set on a
fresh fit; drop resume_from or revert them.

And with an identity-breaking change, the derivative source (derivatives="autograd", the config field use_autograd):

ValueError: resume config mismatch for algorithm 'sga': identity_breaking field(s)
changed vs the checkpoint: use_autograd (stored False -> live True). A science-identity
change starts a new lineage: run a fresh fit in a new checkpoint directory; this
checkpoint cannot be resumed under the new config.

Every message names the field and both of its values.

Add generations

generations is not a config field. For the iterative algorithms it maps to the iteration budget, so extending a run means raising generations on the resuming model. PySR stores its populations and hall of fame in the checkpoint, so raising generations on a resume extends that segment's PySR search. PySINDy solves in one pass, so for it generations cannot be raised on a resume.

The two manifests

The manifest.json above belongs to a checkpoint directory and lists what is in it. A finished run carries a second, unrelated manifest at model.result_.manifest. It records the facts needed to reproduce that run and excludes anything that varies between two identical runs: the dataset fingerprint, the KD version, the seed, the term library for the algorithms that take one, and, when the run continued an earlier one, where it resumed from.