Performance SLOs
Jammi’s performance contract is throughput and coverage, gated against committed baselines — not latency. Each scale-relevant engine verb commits a measured rate (or, for the recall tier, a recall fraction gated against a committed floor) on a named reference box, and a regression gate fails when a fresh run falls more than a fixed fraction below it. This page is the operator’s reference for every gated target: the verb, the named scale it is measured at, the committed baseline, the relative-drop threshold, and the box the baseline was emitted on.
How the gate works
A measured rate must not fall more than the relative-drop threshold below
its committed baseline. The threshold derives an absolute floor from the
baseline — floor = baseline · (1 − threshold) — and the gate is a >= against
that floor (measured >= floor), never an equality and never a bit-compare. The
single threshold is 30% (DEFAULT_REGRESSION_THRESHOLD), defined once in the
harness. It is generous on purpose: the load-bearing failure this gate exists to
catch is a structural regression — an algorithm that went quadratic, a lock
that serialized a parallel path, a dropped fast path — which collapses
throughput by far more than a third. A tighter threshold would trade that real
signal for false alarms on runner noise.
The gate fails closed: a non-finite or non-positive baseline cannot anchor a
relative gate, so it fails (it never vacuously passes against a meaningless
baseline). Each *-scale bench subcommand maps its verdict to its process exit
code — a regression exits non-zero — which is what the CI lanes assert.
Where the gate runs
| Lane | Trigger | Blocking? | Purpose |
|---|---|---|---|
ci.yml (workspace tests) | every PR | yes | Gates a property of the mechanism: committed_baseline_gates_with_teeth proves the committed baseline is a well-formed, generously-thresholded gate that can fail. It does not re-measure the rate on the contended PR runner. |
perf.yml | nightly schedule: + workflow_dispatch | no (early-warning) | Runs every *-scale tier’s measured-rate gate on a real box, so a structural regression surfaces between releases. Non-blocking because the 30% band was sized for a same-box manual emit, not a contended shared runner — a required per-PR rate gate would flap and rot. |
crates.yml (perf-gate) | v* release tag | yes | The authoritative same-box-ish gate: publish depends on it, so a structural perf regression on the release tag blocks the crates.io publish and the GitHub release. |
The gated targets
Each row is one gated verb at one named scale. The rates are same-box
throughputs; the recall row is a fraction gated by an inequality — the
measured >= floor check is meaningful on any box, but the fraction itself is
bit-for-bit only on the same box (see the same-box caveat). Every committed number
is a real, re-derivable fold — a rebuild-* bench subcommand reproduces it on
the emit box.
| Verb | Bench tier | Named scale | Committed baseline | Threshold | Gated quantity |
|---|---|---|---|---|---|
fine_tune | train-scale | 1 536 in-batch-negative pairs, one GradCache backward + AdamW step, Device::Cpu | 180.0 pairs/s | 30% rel. drop | throughput (pairs/s) |
fine_tune_graph | graph-train-scale | 8 communities × 64 nodes, biased-walk sampler (walk length 4, 4 walks/node) | 6 418.1 pairs/s | 30% rel. drop | sampled-pairs/s throughput (+ a portable determinism digest) |
train_context_predictor | context-predictor-scale | CNP over 8 tasks × 18 rows, 30 epochs | 21.29 episode-steps/s | 30% rel. drop | meta-training throughput (+ a same-box predict digest) |
generate_embeddings | model-inference-scale | 16 rows over a tiny 32-dim 1-layer BERT bundle, Device::Cpu | 333.6 rows/s | 30% rel. drop | coarse serving throughput (+ a same-box embed digest) |
infer (classification) | model-inference-scale | 16 rows over a tiny 32-dim 1-layer ModernBERT classifier bundle, Device::Cpu | 207.0 rows/s | 30% rel. drop | coarse serving throughput (+ a same-box infer digest) |
search + build_neighbor_graph | arxiv | 2 000-row corpus slice, 100 held-out 768-dim queries (frozen sidecar) | recall@{1,10,100} = {1.0, 1.0, 0.997} | floor = measured − 0.04 (absolute margin) | recall fraction (not a rate) — measured >= floor, an inequality gate whose absolute margin absorbs cross-box float drift; the fraction is bit-for-bit only on the same box |
The reference box
The committed rate baselines were emitted on this box, in the release
profile, with RAYON_NUM_THREADS=1:
| Property | Value |
|---|---|
| Logical CPUs | 8 |
| Total RAM | 31 720 MiB (~31 GiB) |
| Profile | release |
| Engine version when committed | 0.30.0 |
A baseline is refreshed by hand (via the tier’s rebuild-* subcommand) when the
emit box changes; the version-stamped report lets a downstream gate reject a
cross-version comparison.
The same-box caveat
A committed rate is not a portable floor. Stated verbatim from the gate’s own definition:
A rate (throughput, QPS, pairs/s) is not portable the way the recall fraction is — it is a property of the box that produced it, so a committed rate baseline is a same-box reference, refreshed by hand when the emit box changes, not a number a different machine can re-derive.
What stays portable is the shape of the gate (a measured rate must not fall
more than a fixed fraction below the committed baseline; a measured recall must
not fall below the committed floor) — that is the sense of “portable” in the
quote above: the floor travels to another box, not the bits. Of the digests
above, only the fine_tune_graph sampled-pair-set checksum is portable
bit-for-bit: the pair selection is a seeded integer stream (its scalar f64
roulette arithmetic is neither contracted nor reordered by Rust) and the
checksum is an FNV-1a fold over the selected node-text bytes, so any box
re-derives it exactly. The recall fraction is not in that class — it is
scoped like the float digests. Recall-set membership is decided by an f32
cosine reduction (the exact oracle’s cosine_distance, a sequential f32
accumulation over the dot product and norms), so the fraction is bit-for-bit
only on the same box; across boxes or architectures a near-tie can move a
neighbour in or out of the top-k, and the recall SLO is an inequality gate
(measured >= floor) whose absolute margin (0.04) absorbs that small float
drift — never a bit-for-bit equality. The predict/embed/infer digests fold an
f32 forward, and an f32 reduction is NOT bit-identical across CPUs
(SIMD/FMA contraction and BLAS reduction order differ by machine), so those
three are a same-box property: each is re-derived on the box that ran it, not
asserted equal across boxes. So the
rate rows above are meaningful only against the reference box; do not read
them as a throughput your hardware must hit. The release-tag gate is the
authoritative reading because it runs on a same-box-ish runner; the nightly lane
is early-warning, not a portable promise.
Why no latency SLOs
The contract is throughput and coverage, not latency. A latency SLO on a shared CI runner flaps — tail latency on a contended box is dominated by co-tenant load, not by the engine’s code path — so a latency gate would either flap (set tight) or never bite (set loose), exactly the failure mode the relative-drop rate threshold is designed around. Latency is therefore out of scope here. The representative full-scale serving numbers (the GPU-model rates that latency would ride on) are captured off-box in the cookbook’s A/B split, not gated in CI.