A prediction with a number is half a prediction; the other half is how sure. This chapter audits the uncertainty channel of the committed tier-04 gaussian year predictor — does its stated spread match reality? It runs on CPU against the cache (no GPU, no recompute): the engine’s eval_calibration produces the proper-score and ECE headline, and an independent numpy PIT/reliability fold cross-checks every number and draws the diagnostic the aggregate summarizes.
14.1 The predictive distributions
The cache holds the gaussian context predictor’s per-row pred_mean and pred_std on the test era, paired with the realised true_year. eval_calibration wants a golden source pairing each predictive distribution with its outcome, so we materialize the committed rows into a registered source with the gaussian schema (record_id, mean, sd, outcome).
import tempfileimport numpy as npimport pyarrow as paimport jammifrom jammi_cookbook import contracts, datasets, railsreg = contracts.load_artifact("arxiv.tier04_regression").to_pylist()test = [r for r in reg if r["split"] =="test"]print(f"test-era predictive distributions: {len(test)}")golden = pa.table({"record_id": [r["paper_id"] for r in test],"mean": [float(r["pred_mean"]) for r in test],"sd": [float(r["pred_std"]) for r in test],"outcome": [float(r["true_year"]) for r in test],})golden_url = datasets._write_parquet(golden, "arxiv_calibration_golden")# A fresh artifact dir each run, so re-execution never collides with a# previously-registered source (the catalog persists registrations per artifact dir).db = jammi.connect(f"file://{tempfile.mkdtemp(prefix='jammi_calibration_')}") # CPU embed armdb.add_source("arxiv_calibration_golden", url=golden_url, format="parquet")
test-era predictive distributions: 2115
14.2 The engine’s calibration report
eval_calibration(shape="gaussian") returns the proper-score headline (crps, nll), the calibration diagnostic (adaptive_ece), sharpness, and central coverage. The golden_source is addressed by its full catalog path.
The proper scores are the headline. CRPS (Matheson and Winkler 1976) and NLL (Gneiting and Raftery 2007) are strictly proper: they are optimized in expectation only by the true predictive distribution, so they reward a forecast that is both accurate and honestly spread — you cannot game them by being over-confident. CRPS ≈ 0.39 (in years; lower is better) and NLL ≈ 1.08 summarize the predictor’s distributional quality in one number each.
14.3 Proper scoring, cross-checked
A proper score is only trustworthy if we know what it computes. We recompute CRPS and NLL independently with the gaussian closed forms over the same committed predictions, and confirm they match the engine to the digit.
from math import erf, pimu = np.array([r["pred_mean"] for r in test])sd = np.array([r["pred_std"] for r in test])y = np.array([float(r["true_year"]) for r in test])def std_normal_cdf(z):return0.5* (1.0+ np.vectorize(erf)(z / np.sqrt(2.0)))z = (y - mu) / sdphi = np.exp(-(z **2) /2.0) / np.sqrt(2.0* pi) # standard-normal pdf at zPhi = std_normal_cdf(z)# Gaussian CRPS closed form (Gneiting & Raftery 2007).crps_manual =float(np.mean(sd * (z * (2.0* Phi -1.0) +2.0* phi -1.0/ np.sqrt(pi))))# Gaussian NLL.nll_manual =float(np.mean(0.5* np.log(2.0* pi * sd **2) +0.5* z **2))print(f"manual CRPS {crps_manual:.4f} vs engine {crps:.4f}")print(f"manual NLL {nll_manual:.4f} vs engine {nll:.4f}")
manual CRPS 0.4894 vs engine 0.4894
manual NLL 1.2165 vs engine 1.2165
14.4 The PIT: is the predictive distribution honest?
The sharpest test of calibration is the probability integral transform (PIT): if the predictive distribution is honest, then \(u_i = F_i(y_i)\) — the predictive CDF evaluated at the realised outcome — is uniform on \([0,1]\)(Dawid 1984). A PIT histogram that bulges in the middle means the predictor is under-confident (too wide); one that piles up at the ends means over-confident (too narrow). We quantify the departure with the Kolmogorov–Smirnov statistic against the uniform.
pit = std_normal_cdf((y - mu) / sd)order = np.sort(pit)n =len(order)ecdf = np.arange(1, n +1) / npit_ks =float(np.max(np.abs(ecdf - order))) # KS distance from Uniform(0,1)print(f"PIT KS statistic (departure from uniform): {pit_ks:.3f}")# Reliability (binned PIT): the fraction of outcomes below each predictive quantile.quantile_levels = np.linspace(0.1, 0.9, 9)observed_below = np.array([float(np.mean(pit <= q)) for q in quantile_levels])for q, obs inzip(quantile_levels, observed_below): bar ="#"*int(round(obs *40))print(f" predicted ≤ q={q:.1f}: observed {obs:.3f}{bar}")
The verdict is honest and negative: the PIT is far from uniform (KS ≈ 0.42), and the reliability curve shows the outcomes are not spread across the predictive quantiles as a calibrated forecast would place them. The predictor is sharp but miscalibrated under the time-split — exactly the non-exchangeability the conformal tier reports from the other direction (under-coverage). The proper scores already penalize this (a calibrated predictor of the same sharpness would score better); the PIT localizes it.
14.5 The sharpness-vs-calibration tradeoff
Sharpness and calibration are two distinct virtues, and the order matters: the guiding principle is to maximize sharpness subject to calibration(Gneiting, Balabdaoui, and Raftery 2007). Sharpness (a narrow predictive spread) is desirable only once the forecast is calibrated — a confident-but-wrong forecast is worse than an honest-but-vague one, and a strictly proper score is precisely the instrument that refuses to reward the former.
sharpness (mean predictive σ): 2.760 years
central coverage: 0.878
eval_calibration(shape='gaussian') on the committed tier-04 gaussian year predictions (pred_mean, pred_std, true_year) for the test era. The proper scores (CRPS 0.489, NLL 1.216) and the sharpness (2.76) headline the predictor; the adaptive ECE (0.223) and the PIT KS statistic (0.471) are the calibration diagnostics — the PIT is far from uniform, so the predictive distribution is miscalibrated under the time-split (the same non-exchangeability the conformal tier reports). Sharpness without calibration: the predictor is confident (narrow) but its quantiles are not honest. All values cross-checked against an independent numpy PIT/CRPS/NLL fold over the same committed predictions.
14.6 Bridge note
Uncertainty is a first-class output, and its honesty is measurable. A predictive distribution earns trust through proper scores (CRPS, NLL) that no over-confident forecast can game (Gneiting and Raftery 2007; Matheson and Winkler 1976), and its calibration is read off the PIT — uniform iff the distribution is honest (Dawid 1984) — and summarized by the ECE (Guo et al. 2017). The principle is to maximize sharpness subject to calibration(Gneiting, Balabdaoui, and Raftery 2007): this tier-04 predictor is sharp (≈ 2.9-year spread) but not calibrated (PIT KS ≈ 0.42) under the dataset’s time-split — the same non-exchangeability the conformal chapter reports as under-coverage, here read as a non-uniform PIT. The two views are one finding: an honest engine reports the miscalibration rather than hiding it.
Dawid, A. Philip. 1984. “Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach.”Journal of the Royal Statistical Society: Series A 147 (2): 278–90.
Gneiting, Tilmann, Fadoua Balabdaoui, and Adrian E. Raftery. 2007. “Probabilistic Forecasts, Calibration and Sharpness.”Journal of the Royal Statistical Society: Series B 69 (2): 243–68.
Gneiting, Tilmann, and Adrian E. Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.”Journal of the American Statistical Association 102 (477): 359–78.
Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. “On Calibration of Modern Neural Networks.” In Proceedings of the 34th International Conference on Machine Learning (ICML).
Matheson, James E., and Robert L. Winkler. 1976. “Scoring Rules for Continuous Probability Distributions.”Management Science 22 (10): 1087–96.