Skip to contents

Validation Evidence and Interpretation

JoonHo Lee ()

2026-08-22

Source: vignettes/validation.Rmd
validation.Rmd

What this vignette validates

Validation asks several different questions. They should not be collapsed into one claim that a calibrated value “matches reliability.”

Layer Question Appropriate evidence
Mathematical kernel Are D = 1 3PL probability and Fisher information implemented correctly? Closed forms, finite differences, extreme-predictor tests, external TAM calculations
Root fit Did the algorithm solve its empirical objective? Calibration residual on the calibration quadrature
Population approximation Does the selected scale generalize to new ability draws? Independent holdout samples and sensitivity to quadrature size
Stochastic algorithm Is SAC stable under its finite step schedule and seed contract? Tuning grid, status flags, branch guard, exact replay
Fitted scores What does an external estimator recover from generated responses? Estimator-specific EAP/WLE diagnostics, kept separate from analytic reliability
Practical impact Does targeting alter a downstream analysis in a bounded simulation? Paired outcome pilot with explicit denominators and limitations

This vignette uses only saved Phase 7 evidence dated 2026-08-19. The render mode for this copy is embedded compact fallback (source-package build). The large evidence runners are not executed while building the vignette. Repository builds read the checked-in CSV files; source-package builds, where dev/ is intentionally excluded, use the compact values embedded below. An earlier unbacked 960-condition narrative is not part of the 0.3.0 evidence base.

Validating a result in your own study

Start by checking the result’s estimand, item scope, topology, and status. A small root residual is necessary, but it describes the fitted empirical objective rather than population accuracy.

fit <- eqc_calibrate(
  target_rho = 0.75,
  n_items = 20,
  model = "3pl",
  item_params = list(
    guessing_params = list(distribution = "fixed", value = 0.20)
  ),
  reliability_metric = "info",
  M = 5000,
  seed = 42
)

summary(fit)
fit$item_scope                 # "fixed_form"
fit$misc$selected_root_direction
abs(fit$achieved_rho - fit$target_rho)

both <- compute_rho_both(
  c = fit$c_star,
  theta_vec = fit$theta_quad,
  beta_vec = fit$beta_vec,
  lambda_base = fit$lambda_base,
  theta_var = fit$theta_var,
  guessing = fit$guessing_vec,
  return_diagnostics = TRUE
)
both[c("rho_tilde", "rho_bar")]

For the same finite information distribution, Jensen’s inequality implies \rho_{info} \geq \rho_{MSEM}. The gap is a property of that design and integration measure; universal cutoffs for a “small” or “large” gap are not used here.

EQC and SAC are comparable only when target, metric, model, latent specification, item scope, and realized fixed form agree. An EQC warm start is therefore paired with resample_items = FALSE; if item arguments are omitted, SAC reuses the stored EQC form.

sac_fit <- sac_calibrate(
  target_rho = fit$target_rho,
  n_items = fit$n_items,
  model = fit$model,
  reliability_metric = "info",
  c_init = fit,
  resample_items = FALSE,
  seed = 42
)

compare_eqc_sac(fit, sac_fit)

Analytic and numerical 3PL oracles

The analytic ledger contains nine independent gates and 33 testthat expectations. It covers the exact g=0 reduction, a closed-form information value at \theta=\beta, predictors through \eta=\pm1000, finite-difference derivatives and information, randomized edge grids, Jensen ordering, and direct reconciliation of both reliability definitions.

analytic_fallback <- data.frame(
  check_id = c(
    "g0_exact_reduction", "theta_equals_beta", "extreme_eta",
    "information_order", "finite_difference_derivative",
    "finite_difference_information", "randomized_edge_grid",
    "jensen_order", "reliability_reconciliation"
  ),
  n_cases = c(2424, 6, 7, 65, 400, 400, 2015, 120, 240),
  statistic = c(
    "failed components", "max abs info error", "max abs log-I error",
    "max log(I3)-log(I2)", "max relative derivative error",
    "max relative information error", "max abs log-I error",
    "minimum rho-info minus rho-MSEM", "max abs reliability error"
  ),
  observed = c(0, 2.220446e-16, 2.842171e-14, -9.663381e-13,
               7.614974e-09, 1.523016e-08, 9.094947e-13,
               1.698087e-07, 2.220446e-16),
  passed = TRUE,
  stringsAsFactors = FALSE
)

analytic <- read_cached_evidence("analytic-oracles-v03.csv", analytic_fallback)
analytic_display <- analytic[, c("check_id", "n_cases", "statistic", "observed", "passed")]
analytic_display$observed <- format(analytic_display$observed, digits = 4)
knitr::kable(analytic_display, row.names = FALSE)
check_id n_cases statistic observed passed
g0_exact_reduction 2424 failed components 0.000e+00 TRUE
theta_equals_beta 6 max abs info error 2.220e-16 TRUE
extreme_eta 7 max abs log-I error 2.842e-14 TRUE
information_order 65 max log(I3)-log(I2) -9.663e-13 TRUE
finite_difference_derivative 400 max relative derivative error 7.615e-09 TRUE
finite_difference_information 400 max relative information error 1.523e-08 TRUE
randomized_edge_grid 2015 max abs log-I error 9.095e-13 TRUE
jensen_order 120 minimum rho-info minus rho-MSEM 1.698e-07 TRUE
reliability_reconciliation 240 max abs reliability error 2.220e-16 TRUE

All nine gates passed. The largest finite-difference relative errors were 7.61\times10^{-9} for the derivative and 1.52\times10^{-8} for information. This supports the D = 1 dichotomous kernel on the evaluated finite supports; it is not evidence for multidimensional or polytomous models.

External TAM oracle and estimator boundary

TAM 4.3-25 supplied an external probability/information calculation for one fixed-guessing 3PL fit. The oracle grid contained 61 theta points and eight items: 488 item-level values and 61 test-information values.

external_fallback <- data.frame(
  comparison = c("probability", "item_information", "test_information"),
  n_values = c(488, 488, 61),
  max_abs_difference = c(1.110223e-16, 3.329906e-08, 1.225098e-07),
  tolerance = c(5e-13, 5e-06, 1e-05),
  pass = TRUE,
  stringsAsFactors = FALSE
)

external <- read_cached_evidence(
  "external-3pl-oracle-v03/oracle-summary.csv",
  external_fallback
)
external_display <- external[, c(
  "comparison", "n_values", "max_abs_difference", "tolerance", "pass"
)]
external_display$max_abs_difference <- format(external_display$max_abs_difference, digits = 4)
external_display$tolerance <- format(external_display$tolerance, digits = 3)
knitr::kable(external_display, row.names = FALSE)
comparison n_values max_abs_difference tolerance pass
probability 488 1.110e-16 5e-13 TRUE
item_information 488 3.330e-08 5e-06 TRUE
test_information 61 1.225e-07 1e-05 TRUE

The TAM fit produced finite EAP scores for 300/300 persons. Its EAP score reliability was 0.4949, while the analytic fitted-bank values were 0.5103 for average information and 0.4814 for MSEM. These are different estimands: external EAP reliability depends on the fitted parameters, prior, scoring rule, and response sample. No equality criterion was applied. The bounded fit also reached its configured 30-iteration cap, so item and score recovery summaries remain descriptive.

The exported TAM::tam.wle() call failed for the tam.mml.3pl object with object 'res' not found. No class mutation or undocumented workaround was used. Consequently, the 0.3.0 external 3PL evidence is EAP-only, and the package makes no 3PL WLE claim. The public compute_reliability_tam() helper continues to support its documented Rasch/2PL WLE and EAP paths.

EQC: calibration residual versus independent holdout

The EQC study crossed two models (2PL and fixed-guessing 3PL), two targets (0.65, 0.80), three calibration sizes (M=500, 2000, 8000), and six fixed forms per cell. It produced 72/72 successful calibrations. Each run used 12 independent holdout replicates of 4,000 theta draws, for 864 replicate rows and 3,456,000 holdout theta draws. All 12 grain/seed/denominator QA checks passed.

The maximum same-quadrature calibration residual was 5.17\times10^{-6}. Across the 72 independently evaluated forms, however, holdout MAE was 0.00402 and RMSE was 0.00598. The two quantities answer different questions.

eqc_m_fallback <- data.frame(
  eqc_M = c(500, 2000, 8000),
  n_designed_runs = c(24, 24, 24),
  n_success = c(24, 24, 24),
  holdout_mae = c(0.006400605, 0.003492137, 0.002180036),
  holdout_rmse = c(0.009031318, 0.004319202, 0.002654830),
  mean_calibration_runtime_sec = c(0.1122917, 0.3788333, 1.6102917),
  stringsAsFactors = FALSE
)

eqc_m <- read_cached_evidence(
  "eqc-holdout-v03/eqc-holdout-M-summary.csv",
  eqc_m_fallback
)
eqc_display <- eqc_m[, c(
  "eqc_M", "n_designed_runs", "n_success", "holdout_mae",
  "holdout_rmse", "mean_calibration_runtime_sec"
)]
names(eqc_display) <- c("M", "designed", "success", "holdout_MAE",
                        "holdout_RMSE", "mean_runtime_sec")
eqc_display$holdout_MAE <- round(eqc_display$holdout_MAE, 6)
eqc_display$holdout_RMSE <- round(eqc_display$holdout_RMSE, 6)
eqc_display$mean_runtime_sec <- round(eqc_display$mean_runtime_sec, 3)
knitr::kable(eqc_display, row.names = FALSE)
M designed success holdout_MAE holdout_RMSE mean_runtime_sec
500 24 24 0.006401 0.009031 0.112
2000 24 24 0.003492 0.004319 0.379
8000 24 24 0.002180 0.002655 1.610

MAE and RMSE decreased at all three aggregate anchors, and each of the 12 model-by-target-by-M series improved as M increased. This is a descriptive sensitivity result for the studied normal-latent, 25-item designs, not a universal convergence rate or a rule that M=8000 is sufficient everywhere.

SAC: tuning sensitivity and reproducibility

The SAC evidence contains 72 unique runs: two models, 12 tuning configurations, and three seeds. Sixty-two runs had status ok; ten deliberately stressful 2PL runs were not_converged. No run lost its selected branch. The selected rows below show why a finite SAC result must be reported with its tuning and status.

sac_fallback <- data.frame(
  model = rep(c("2pl", "3pl"), each = 4),
  config_id = rep(c("baseline", "A_high", "gamma_high", "iterations_short"), 2),
  holdout_rmse = c(0.02777352, 0.06079836, 0.06735408, 0.05321610,
                   0.01138192, 0.01541072, 0.02420332, 0.02012300),
  ok_rate = c(1, 0, 0, 1/3, 1, 1, 1, 1),
  branch_lost_rate = 0,
  stringsAsFactors = FALSE
)

sac_all <- read_cached_evidence(
  "sac-sensitivity-v03/sac-sensitivity-summary.csv",
  sac_fallback
)
sac_selected <- sac_all[
  sac_all$config_id %in% c("baseline", "A_high", "gamma_high", "iterations_short"),
  c("model", "config_id", "holdout_rmse", "ok_rate", "branch_lost_rate")
]
sac_selected <- sac_selected[order(sac_selected$model, match(
  sac_selected$config_id,
  c("baseline", "A_high", "gamma_high", "iterations_short")
)), ]
sac_selected$holdout_rmse <- round(sac_selected$holdout_rmse, 4)
sac_selected$ok_rate <- round(sac_selected$ok_rate, 3)
knitr::kable(sac_selected, row.names = FALSE)
model config_id holdout_rmse ok_rate branch_lost_rate
2pl baseline 0.0278 1.000 0
2pl A_high 0.0608 0.000 0
2pl gamma_high 0.0674 0.000 0
2pl iterations_short 0.0532 0.333 0
3pl baseline 0.0114 1.000 0
3pl A_high 0.0154 1.000 0
3pl gamma_high 0.0242 1.000 0
3pl iterations_short 0.0201 1.000 0

Baseline holdout RMSE was 0.0278 for 2PL and 0.0114 for 3PL in these two representative forms. High A, high \gamma, and a 50-iteration budget were materially worse for the 2PL fixture; high \gamma and the short budget also increased 3PL error. These values do not rank settings universally.

Exact same-seed replay passed for both models: c_star, scale and reliability trajectories, achieved distributions, and caller RNG restoration were identical in the same R/RNG environment.

Guessing, topology, and heavy tails

The empirical map crossed three guessing levels, two targets, two test lengths, three item-coverage designs, and three latent shapes, yielding 108 cells. All were feasible and resolved with one root. Within the 72 matched contrasts, increasing guessing never decreased the selected scale.

guess_fallback <- data.frame(
  guessing = rep(c(0, 0.15, 0.30), 2),
  target_rho = rep(c(0.60, 0.80), each = 3),
  n_cells = 18,
  feasible_rate = 1,
  multiple_root_rate = 0,
  median_selected_c = c(0.7822571, 1.1510567, 1.4757482,
                        1.7589756, 2.4765858, 3.3040362),
  stringsAsFactors = FALSE
)

guess_summary <- read_cached_evidence(
  "guessing-topology-v03/guessing-topology-summary.csv",
  guess_fallback
)
guess_display <- guess_summary[, c(
  "guessing", "target_rho", "n_cells", "feasible_rate",
  "multiple_root_rate", "median_selected_c"
)]
guess_display$median_selected_c <- round(guess_display$median_selected_c, 3)
knitr::kable(guess_display, row.names = FALSE)
guessing target_rho n_cells feasible_rate multiple_root_rate median_selected_c
0.00 0.6 18 1 0 0.782
0.15 0.6 18 1 0 1.151
0.30 0.6 18 1 0 1.476
0.00 0.8 18 1 0 1.759
0.15 0.8 18 1 0 2.477
0.30 0.8 18 1 0 3.304

The typical map does not prove monotonicity. A separate 16-cell, seven-node adversarial fixture contained two or four roots in every cell, with both increasing and decreasing branches. This is why topology diagnostics and the default lowest_increasing policy remain part of the calibration contract.

For standardized Student-t_5 prefixes at M=100{,}000, the finite-prefix MSEM reliability was 0.001514 for g=.20 and 0.000884 for g=.30, while the corresponding information reliabilities were 0.6695 and 0.6177. Finite values do not establish population integrability. The built-in heavy_tail plus population MSEM target remains a classed error; use information reliability or define an explicit finite/truncated empirical estimand.

Downstream DIF and score-recovery smoke evidence

The primary practical pilot used one bounded mechanism: D = 1 3PL, fixed guessing 0.20, 20 items, 500 persons per group, 20 paired replications, and three arms (c=1, target 0.60, target 0.80). Null and four-item DIF conditions shared item, person, and uniform random streams across arms. The evidence contains 60 calibration rows, 400 fixed-form item rows, 120 replicate-arm- condition rows, and 2,400 item-level Mantel–Haenszel rows. All 17 blocking QA checks passed; the maximum independent holdout error for targeted arms was 0.0171.

dif_fallback <- data.frame(
  arm = rep(c("uncontrolled_c1", "target_rho_060", "target_rho_080"), each = 3),
  metric = rep(c("null_type_i", "dif_power", "dif_nondif_type_i"), 3),
  n_replicates = 20,
  total_rejections = c(20, 43, 21, 16, 30, 13, 9, 59, 27),
  total_denominator = rep(c(400, 80, 320), 3),
  mean_replicate_rate = c(0.0500, 0.5375, 0.065625,
                          0.0400, 0.3750, 0.040625,
                          0.0225, 0.7375, 0.084375),
  mean_holdout_rho_info = rep(c(0.7049195, 0.5979468, 0.7976339), each = 3),
  stringsAsFactors = FALSE
)

dif_summary <- read_cached_evidence(
  "practical-impact-v03/dif-summary.csv",
  dif_fallback
)
dif_display <- dif_summary[, c(
  "arm", "metric", "total_rejections", "total_denominator",
  "mean_replicate_rate", "mean_holdout_rho_info"
)]
dif_display$mean_replicate_rate <- round(dif_display$mean_replicate_rate, 4)
dif_display$mean_holdout_rho_info <- round(dif_display$mean_holdout_rho_info, 4)
knitr::kable(dif_display, row.names = FALSE)
arm metric total_rejections total_denominator mean_replicate_rate mean_holdout_rho_info
uncontrolled_c1 null_type_i 20 400 0.0500 0.7049
uncontrolled_c1 dif_power 43 80 0.5375 0.7049
uncontrolled_c1 dif_nondif_type_i 21 320 0.0656 0.7049
target_rho_060 null_type_i 16 400 0.0400 0.5979
target_rho_060 dif_power 30 80 0.3750 0.5979
target_rho_060 dif_nondif_type_i 13 320 0.0406 0.5979
target_rho_080 null_type_i 9 400 0.0225 0.7976
target_rho_080 dif_power 59 80 0.7375 0.7976
target_rho_080 dif_nondif_type_i 27 320 0.0844 0.7976

Paired DIF-power differences were -0.1625 for target 0.60 minus c=1 (20-replicate t interval [-0.2497, -0.0753]) and +0.2000 for target 0.80 minus c=1 ([0.0887, 0.3113]). Within this prespecified mechanism, discrimination/reliability scaling altered detection. It does not establish that higher target reliability always improves operating characteristics: null rejection and rejection among non-DIF items also changed.

The secondary TAM arm contained three fits per arm (nine total), 180 item rows, 2,700 person rows, and 549 information-curve rows. Mean EAP correlations for c=1, target 0.60, and target 0.80 were 0.834, 0.735, and 0.883; mean EAP RMSEs were 0.544, 0.666, and 0.463. Eight of nine fits reached the 30-iteration cap, so recovery is exploratory.

This is a partial response to the reviewer’s practical-significance criticism, not a resolution. The pilot has only 20 DIF replications, one test length, one guessing level, one DIF magnitude, internal-only MH validation, and no purification. Moreover, a fixed difficulty shift of 0.50 changes its response-logit magnitude when the global discrimination scale changes, so the contrast is a paired scale intervention rather than a pure causal effect of a reliability label. Publication-scale Type-I/power claims require a larger factorial study and external DIF implementation.

Runtime and memory diagnostics

The performance artifact contains 36 kernel runs, six calibration runs, four response simulations, and four fresh-process memory runs. All ten performance gates passed. The largest kernel time was 0.541 seconds in the recorded environment.

For M=200{,}000 and 50 3PL items, automatic chunking reduced median process maximum RSS from 1,570.4 MiB to 209.5 MiB (86.7%). Median reducer time was 1.284 seconds unchunked and 1.044 seconds with automatic chunks; all four subprocesses returned the identical parsed reliability, 0.8695789. These are macOS arm64, R 4.6.0 diagnostics, not cross-platform service-level guarantees.

Phase 7 acceptance ledger 

ledger_fallback <- data.frame(
  phase_step = c("7.1", "7.2", "7.3", "7.3", "7.4", "7.4",
                 "7.5", "7.5", "7.5", "7.6", "7.7"),
  observed = c(
    "9/9", "6/6", "72/72 runs; 12/12 QA",
    "MAE .006401 -> .003492 -> .002180", "72 unique runs",
    "2/2 model replays", "108 resolved; 72 paired deltas", "2/4 roots",
    "max-M g>0 MSEM max=.001514", "17/17; DIF rows=2400",
    "10/10; RSS reduction=86.7%"
  ),
  pass = TRUE,
  caveat = c(
    "D=1 dichotomous, finite supports",
    "TAM fit capped; WLE unsupported",
    "normal latent; six forms/cell",
    "three-anchor descriptive trend",
    "one form/model; three seeds",
    "same R/RNG environment",
    "empirical sensitivity map",
    "adversarial fixture",
    "population MSEM nonintegrable",
    "20-replicate smoke; partial response",
    "single macOS arm64 environment"
  ),
  stringsAsFactors = FALSE
)

ledger <- read_cached_evidence("phase7-acceptance-v03.csv", ledger_fallback)
ledger_display <- ledger[, c("phase_step", "observed", "pass", "caveat")]
knitr::kable(ledger_display, row.names = FALSE)
phase_step observed pass caveat
7.1 9/9 TRUE D=1 dichotomous, finite supports
7.2 6/6 TRUE TAM fit capped; WLE unsupported
7.3 72/72 runs; 12/12 QA TRUE normal latent; six forms/cell
7.3 MAE .006401 -> .003492 -> .002180 TRUE three-anchor descriptive trend
7.4 72 unique runs TRUE one form/model; three seeds
7.4 2/2 model replays TRUE same R/RNG environment
7.5 108 resolved; 72 paired deltas TRUE empirical sensitivity map
7.5 2/4 roots TRUE adversarial fixture
7.5 max-M g>0 MSEM max=.001514 TRUE population MSEM nonintegrable
7.6 17/17; DIF rows=2400 TRUE 20-replicate smoke; partial response
7.7 10/10; RSS reduction=86.7% TRUE single macOS arm64 environment

All 11 integrated blocking gates passed. Their caveats define the boundary of the supported claim rather than optional footnotes.

Confidence assessment

Claim Assessment Required qualification
D = 1 unidimensional 3PL probability/information kernel Ready to share Finite-support, dichotomous scope
EQC fixed-form information calibration Ready to share Report root topology and separate root residual from independent holdout error
SAC calibration Share with caveats Report metric, item scope, tuning, status, branch diagnostics, and independent achieved distribution
TAM 3PL EAP diagnostic Share with caveats Estimator-specific; bounded fit reached cap; WLE unsupported
DIF practical significance Needs a larger study for a publication-scale claim Current evidence is a paired 20-replicate mechanism smoke only
Performance Ready as an environment-specific regression diagnostic Not a portable runtime or memory guarantee

Evidence provenance and fingerprints

The repository runners and their SHA-256 fingerprints are:

26da32f9587371cf4e20e5cecfa397916c08021e82a30b6af8f13751598bf76f  dev/evidence/analytic-oracles-v03.R
feea7961ea48b6d2ff2a069e333e796f728dc112b23460c104bb8c72e854c094  dev/evidence/external-3pl-oracle-v03/run_external_3pl_oracle.R
5fbd6d9917d46b20f9e8540e40eaba46d7a3803f6dff6b839d4b4ea7fa11ccfd  dev/evidence/eqc-holdout-v03.R
080d6635757ebb47c1fa5bbeccd4ec5cf4bd9d43b48d31ecd1494492d0dcbfb7  dev/evidence/sac-sensitivity-v03.R
cce5fc07a7f8a32fd48f100470e034e4c38d90182b58be7bb8e0730dcadccb1a  dev/evidence/guessing-topology-v03.R
f42784e9ab95a71af5d432fa2fa300f7d5fd33df5fba05a6e294dbc34f8268a6  dev/evidence/practical-impact-v03/run_practical_impact_pilot.R
7b2d1a6fa6530c415ac8e432b0bf79a691bcf53c595576ee47e92c81b29312db  dev/evidence/performance-acceptance-v03/run_performance_acceptance.R
daf3455071add89198863d042fa54147bfb342380f4df3918b927f67516d9ecd  dev/evidence/phase7-acceptance-v03.R

Key displayed CSV fingerprints are:

0af0decb2dc258d5ac30c8954e18ce0a3a9671cf20cf18fad82b75b7d2a60293  dev/evidence/analytic-oracles-v03.csv
d37da8595b2e98141cc5cc7690c0186b7b3c6c9a10e7fcb38fd5a4dd63a2840a  dev/evidence/external-3pl-oracle-v03/oracle-summary.csv
c75446ccd165a42017d9cd37bfe485234420e68fd35bd3b9b6b528df4b760d89  dev/evidence/eqc-holdout-v03/eqc-holdout-M-summary.csv
120d11e056753a587463f37300845c00a80a46b40da41b297f55067a5c17a1ba  dev/evidence/sac-sensitivity-v03/sac-sensitivity-summary.csv
06a9dd7d52f4635166e59b0226b8e1fc481715e4c5e53c5c9603d3eedb064aa1  dev/evidence/guessing-topology-v03/guessing-topology-summary.csv
3fe8c81a0466911b8fcfdd0bef7b6d4cede628b5d12de2155f83d25bd06c361d  dev/evidence/practical-impact-v03/dif-summary.csv
d4315ed43c362750619897f3d910c3dd0b3097413a76b7fdd7312186247681f8  dev/evidence/practical-impact-v03/qa-summary.csv
855d94498e7b5662244822256c5eed2f5230c25a5ae6f8e83feb2466126cca6e  dev/evidence/performance-acceptance-v03/performance-acceptance.csv
e3d1206e4f41537fe9e551119ba2bb659fd5db0f35c0bfd0f23cb958562ad417  dev/evidence/performance-acceptance-v03/memory-benchmark.csv
e7cd81c53a8653786c4db941073bffece683fda6483876f67d797b4506d954af  dev/evidence/phase7-acceptance-v03.csv

The external and practical run manifests record R 4.6.0 on aarch64-apple-darwin23, TAM 4.3-25, seeds, row grains, and run dates. They show package source version 0.2.0 because evidence was generated before DESCRIPTION was bumped to 0.3.0; the D = 1 3PL source implementation was already present. Re-run the runners and regenerate fingerprints if production R code changes after this evidence freeze.

Reporting checklist

For a reliability-targeted simulation, report at least:

  1. model and D constant; item guessing generator for 3PL;
  2. reliability metric (info or msem) and latent variance/measure;
  3. fixed-form versus item-superpopulation scope;
  4. global scale bounds, selected root policy, root count, and branch direction;
  5. calibration sample size, root residual, and an independent holdout summary;
  6. SAC tuning, iteration budget, status flags, and achieved distribution when SAC is used;
  7. seeds and item/person resampling design;
  8. external estimator and scoring rule without treating fitted-score reliability as the analytic target;
  9. exact replication denominator and uncertainty for downstream outcomes;
  10. limitations that match the evidence grain.

This separation makes a result auditable: mathematical correctness, numerical root fit, population approximation, fitted-score behavior, and practical impact remain related but distinct validation claims.