# Protocol fixed before the first run

Date: 2026-11-13. Seed: 20261113. These are small public-data benchmark experiments, not clinical validation. No patient upload or medical advice is provided. Data were not collected by Brandon Huang. Analysis and writing were generated with AI assistance for his research notebook.

## 1. Confidence is not accuracy

Wisconsin Diagnostic Breast Cancer, 569 rows, 30 numerical cell-nucleus features derived from fine-needle aspirates. Malignancy is the positive class. Stratified fixed 60/20/20 train/calibration/test partition (rounding follows sklearn). Train StandardScaler and logistic regression C=1 only on training. Freeze classifier, fit sigmoid calibration on calibration set only. Evaluate raw, calibrated, and constant training-prevalence predictions on untouched test. Primary: Brier score with paired difference bootstrap interval. Secondary: AUROC, log loss, confusion matrix, five fixed equal-width reliability bins. Preset confidence thresholds .50, .60, .70, .80, .90, .95, .99; show retained N, deferred N, retained accuracy, and false negatives. No threshold selection on test. This does not prove abstention is safe or that calibration improves deployment.

## 2. A prediction needs room to be wrong

Diabetes progression benchmark, 442 rows, 10 baseline features. Load raw unscaled variables; avoid the dataset's default global standardization. Random fixed 60/20/20 train/calibration/test. Primary predictor: train-only scaling plus ridge alpha=10. Comparator: histogram gradient boosting, 150 iterations, 8 max leaves, L2=10, no early stopping. Training-mean baseline. No tuning or model choice on test. Calibration absolute residuals produce constant-width symmetric split-conformal intervals at 80%, 90%, 95%. Quantile rank ceil((n_cal+1)*(1-alpha)); never interpolate this quantile. Evaluate MAE, RMSE, R², marginal test coverage and width. Exploratory coverage strata use training median BMI, not outcome-based cutoffs. Coverage is marginal under exchangeability, not guaranteed per patient, subgroup, or shifted population.

## 3. A model meets missing information

UCI heart-failure cohort, 299 rows, fixed stratified 70/30 train/test. DEATH_EVENT is the target, with heterogeneous follow-up. Exclude follow-up duration `time` from usable baseline predictors. This is a retrospective binary benchmark, not a fixed-horizon risk predictor or survival model. Median imputation + all-column missingness indicators + StandardScaler + logistic C=1, trained only on training. Compare to training-only augmentation with original rows plus three copies with independent 25% missing cells; no test rows duplicated into training. Each of four copies gets weight 0.25 to preserve total training weight and logistic regularization strength. Both estimators have all-column indicators; the baseline indicators are constant in clean training. Augmentation is a combined missingness-exposure/indicator-learning recipe, not a causal identification claim. This weight/indicator correction was made during independent code audit before interpreting results. Apply paired independent MCAR masks to held-out predictors at 0%, 10%, 25%, 50%, 30 repetitions each. Show mean and 5th–95th percentile of AUROC/Brier over corruption masks, not population confidence intervals. Use identical masks for both models. Deliberately unusable comparator includes `time` only to illustrate retrospective information leakage; never recommend it for prospective prediction.

## Uncertainty and limitations

2000 seeded test-row bootstrap replicates for AUROC/Brier/MAE, percentile intervals; paired bootstrap for Brier difference; Wilson intervals for observed proportions. These intervals condition on one fitted model and one split, not training variability. Masks reflect artificial corruption only. No external sites, temporal split, prospective cohort, treatment effect, clinical utility, fairness conclusion, or validation of imaging performance. These are pre-extracted features, not image-reading models. Source datasets lack site/group identifiers in the distributed analysis tables; no patient-level grouping beyond one row per released record can be verified. No test metrics were used to change hyperparameters or selected thresholds after the initial protocol.
