Skip to content

Score conventions and comparability

KD does not rank algorithms against one another. It provides a way to refit candidate terms against the same data so that the resulting numbers can be compared.

model.best_score_ is that algorithm's own objective value. Its name and its direction are declared per algorithm: some report AIC, some report NMSE, and some report a reward they define themselves, with both min and max directions in use. The score cell in the spec strip at the top of each algorithm page reports these two facts. Equal score values from two different algorithms therefore do not indicate equal quality.

model.result_.score_kind and score_direction expose those two facts programmatically. score_kind is a stable identifier. Two scores are comparable only when the kind, the dataset and the scorer all match, and across algorithms these conditions generally do not hold.

Refit against the same data

kd.evaluate_terms takes a set of terms back to the same data and fits them once, using the same derivatives and the same least-squares solver. This requires one additional step compared with converting the native scores, and it produces a number that refers to a specific fit rather than to a conversion. When the term sets under comparison are all evaluated this way, the resulting nmse, r2 and coefficients are measured in the same frame.

import kd

dataset = kd.load_kdv()
for terms in [
    ["mul(u, u_x)", "diff2_x(u)"],
    ["mul(u, u_x)", "diff_x(diff2_x(u))"],
]:
    result = kd.evaluate_terms(dataset, terms)
    print(terms)
    print("  nmse", format(result.nmse, ".3e"),
          "coefficients", [round(float(c), 4) for c in result.coefficients])

['mul(u, u_x)', 'diff2_x(u)']
  nmse 8.489e-01 coefficients [-0.0902, -0.0024]
['mul(u, u_x)', 'diff_x(diff2_x(u))']
  nmse 2.903e-05 coefficients [-1.0, -0.0025]

The two sets differ in a second versus a third derivative. The second set is the true support of KdV: the refit returns -1.0 and -0.0025, matching the reference equation, at an nmse more than four orders of magnitude lower. These two nmse values can be compared with each other, because both fits used the same data, the same derivatives and the same solver.

Besides nmse, mse, r2 and coefficients, the returned EvaluationResult carries condition_number, the condition number of the columns this fit was handed, which indicates whether the candidate terms are collinear.

The comparison figure

kd.VizEngine.render_comparison overlays several runs: their score curves, a bar chart of final scores and R², and a summary table carrying each run's expression, nmse, r2 and how its term set differs from the first run's.

The mean and standard deviation band over those curves is drawn only across runs that share one algorithm and one declared score kind and direction. If a run reporting AIC is overlaid with one reporting a reward, the band is dropped, a subtitle on the figure states that the curve trends are not comparable, and the accompanying warning points to the final nmse and r2 instead. Two different algorithms that report the same metric also lose the band, because a single band over two populations does not describe either of them.

Accepted term forms

Each string is a single term in function-call form (see Equation representation). Infix notation and a top-level sum are rejected, and the rejection carries the reason. kd.validate_terms classifies terms without fitting them, so it can be used as a first pass:

report = kd.validate_terms(dataset, ["u + u_xx", "add(u, u_xx)", "mul(u, u_x)"])
print(report.valid)
for rejection in report.rejected:
    print(rejection.term, "|", rejection.reason)

['mul(u, u_x)']
u + u_xx | syntax: non-canonical term (funcall IR required); Infix operators (e.g., 'a + b') are not allowed. Use function-call IR: add(a, b), mul(a, b), etc.
add(u, u_xx) | composite: top-level additive structure; submit as 2 separate terms: ['u', 'u_xx']

kd.evaluate_terms is strict by default. One rejected term raises InvalidTermsError carrying the complete list of rejections rather than only the first one. Pass skip_invalid=True and the rejected terms are dropped, the remaining terms are fitted, and one warning names what was dropped.

Terminology

model.result_.run_record.evidence.headline_coefficient_source records who performed the fit (native for coefficients the algorithm produced itself, platform_refit for coefficients the platform refitted). It qualifies coefficients, mse, nmse and r2 only, not the score. Score semantics are governed by score_kind.