Appendix B — Metric and threshold definitions

This appendix states the output-family metrics precisely enough to reimplement, together with the conventions that make independent recomputation land on the same decimals. All of them are pinned to the simulation volume’s own implementations by a verification check in the frozen layer.

B.1 F1, individual scores

For estimate sets a and b over the same n persons, the standardized movement is d_i = (a_i - b_i) / s, where s is the standard deviation of the default report (the Gaussian arm’s PM ensemble) for that case-cell; the same s standardizes every contrast within the cell, so movements are comparable across contrasts. Reported: the median, 90th percentile and maximum of |d_i|, the signed mean, the Pearson and Spearman correlations, and the shares of |d_i| above 0.10, 0.25 and 0.50. Material when the median exceeds 0.10.

B.2 F2, the reported distribution

The between-set distance is the Kolmogorov–Smirnov statistic between the two right-continuous empirical distribution functions, evaluated on the union of the two sorted supports (a distance between reported ensembles; deliberately not a hypothesis test, so no null calibration applies). Alongside it: the SD ratio, IQR ratio, skewness, excess kurtosis, and seven percentiles (5, 10, 25, 50, 75, 90, 95) per set. Material when the KS exceeds 0.05 or the SD ratio leaves [0.95, 1.05].

B.3 F3, rankings and selection

Ranks use average ties; percentile ranks are 100\,r_i/n. Reported: Spearman and Kendall correlations, the median percentile-rank movement, shares moving more than 1, 2 and 5 points, and exact top-k overlaps for 5, 10 and 25% as Jaccard indices J, with k=\lceil pn\rceil and person ID as the deterministic boundary tie-break. The share of the selected group that changes membership is 1 - 2J/(1+J) (so J = 0.818 means 10% of the group changed; the conversion mistake this formula repairs is recorded in Appendix D). Material when more than 5% move more than 2 points or the top-decile Jaccard falls below 0.90.

B.4 F4, tails and cuts

At fixed cuts \theta = 1.0, 1.5, 2.0: each set’s share beyond the cut, the signed difference in percentage points (net), and the share of persons who cross in either direction (reclassification traffic; the two are different quantities and the distinction carries Chapter 13). At percentile cuts (top and bottom decile): raw agreement and Cohen’s \kappa against the ~82% agreement that random selection of the same sizes would produce. Material when any fixed cut’s net change exceeds 2 points.

B.5 Conventions

Numeric quantiles are type 8; top-group membership uses the exact-k rule above; ranks average ties; EDFs are right-continuous on the union grid; the KS argmax reports its first maximizer; a clump is a maximal run of sorted estimates with no gap above 0.02 on the \theta scale. The paired seed reference is measured, not modeled: every statistic above is recomputed on the seed-B replicate, and the Monte Carlo standard error of a statistic is \sqrt{\smash[b]{\overline{d^2}/2}} over the seed pair differences, the two-replicate estimator. The two-MCSE rule (a difference under twice the pooled MCSE is treated as sampler noise) was armed before fitting and never bound: measured MCSEs are of order 0.001 to 0.003 SD, and every reported class-level movement runs at least an order of magnitude above them, which is why materiality, not distinguishability, is the operative bar everywhere in this book.