Operability

How to run a Jammi server in production: what it exposes for observability, how it shuts down cleanly, the resource limits it enforces, and how it behaves when a dependency fails. Everything below describes the system as it ships today.

Observability surface

The server exposes three HTTP side-channel endpoints, independent of the gRPC and Flight SQL data paths:

/healthz answers without touching any dependency, so an orchestrator uses it to decide whether to restart the container. /readyz goes one step further and pings the catalog — a transient catalog outage returns 503 so the load balancer removes the instance from rotation rather than restarting it.

$ curl -s localhost:8080/healthz
{"status":"ok","version":"0.29.0"}

$ curl -s localhost:8080/readyz          # catalog reachable
{"status":"ready"}

$ curl -s localhost:8080/readyz          # catalog unreachable → 503
{"status":"not_ready","detail":"catalog ping failed: connection refused"}

Metrics

/metrics emits these substrate-level metrics that the gRPC services, Flight SQL layer, and request-bounds layer stack feed:

# HELP jammi_grpc_requests_total Total number of gRPC requests served across all jammi.v1 services.
# TYPE jammi_grpc_requests_total counter
jammi_grpc_requests_total 1432
# HELP jammi_search_latency_seconds Vector-search request latency, in seconds.
# TYPE jammi_search_latency_seconds histogram
jammi_search_latency_seconds_bucket{le="0.05"} 311
jammi_search_latency_seconds_bucket{le="0.1"} 402
jammi_search_latency_seconds_sum 27.41
jammi_search_latency_seconds_count 418

Tracing

The server installs a global tracing subscriber (a Registry layered with the fmt formatter and, when configured, an OTLP export layer). Spans carry the correlation fields that let you follow a request across the gRPC surface and the worker fleet:

• gRPC handler spans carry tenant_id — recorded once the handler has resolved the request’s tenant scope.
• run_claimed_job (the worker dispatching a claimed job) carries worker_id, job_id, and tenant_id.
• run_spec (the training run inside a claimed job) carries job_id and worker_id.

Logs are emitted as structured records, JSON or human-readable text per logging.format (LogFormat). The filter comes from logging.level, with RUST_LOG as an optional override. Output always goes to stdout — a server runs non-interactively by design — and ANSI colour is enabled only when stdout is a terminal.

{"timestamp":"2026-06-15T04:17:33.114Z","level":"INFO","fields":{"message":"job completed"},"target":"jammi_ai::fine_tune::worker","span":{"job_id":"job-7af3","worker_id":"worker-2","tenant_id":"acme","name":"run_claimed_job"}}

OTLP trace export

Setting [observability] otlp_endpoint sends spans to any vendor-neutral OTLP collector over gRPC (opentelemetry-otlp, tonic transport). Every span from this process carries the service.name resource attribute (default "jammi"), and the parent-based ratio sampler keeps sample_ratio (default 1.0, i.e. everything) of the traces this process ROOTS — a span whose parent was already sampled by the caller is always kept, regardless of the local ratio. Request headers a collector requires (an auth token, a tenant header) go under [observability.otlp_headers]; each value is a secret — inline or { file = "…" } — and is never logged (see Configuration).

Leaving otlp_endpoint unset installs no exporter and opens no network connection for tracing at all — the zero-egress default. The exporter lives behind the telemetry-otlp cargo feature (on by default in every published jammi-server build and in the jammi-python embed wheel); a build compiled WITHOUT the feature refuses to start with a typed configuration error if otlp_endpoint is set, rather than silently dropping every span.

A whole-server tower layer extracts the incoming W3C traceparent (and tracestate) from every gRPC/Flight request’s HTTP headers and continues that trace: the span this process opens for the request shares the CALLER’s trace id, so a request that entered through an edge proxy or gateway stays one trace end-to-end across process boundaries. A request with no traceparent header starts a fresh, unparented trace, exactly as today.

Graceful shutdown

run_with_shutdown drives both the HTTP side-channel and the gRPC surface and drains them in parallel: the call returns once both have stopped accepting new connections and finished serving in-flight requests. The standalone binary wires both SIGINT (Ctrl+C) and SIGTERM, so docker stop — which sends SIGTERM — triggers the same clean drain as an interactive Ctrl+C.

Backpressure and resource limits

The engine enforces these limits. There is no in-memory worker-queue-depth bound — the work queue is durable, not buffered (see below) — but the combined gRPC + Flight SQL surface (and, separately, the request-bounds knobs below) DOES enforce a configured inbound message-size cap, request-concurrency bounds, and two long-lived-stream budgets.

Request bounds ([server.limits])

Every inbound request to the combined gRPC + Flight SQL listener (and, for message size, the Flight-only listener too) is checked against [server.limits] before any tenant-scoped catalog read runs — a refused request never reaches a handler, so a refusal leaks nothing about cross-tenant existence. A request that would exceed any of these is refused at the edge with a typed gRPC status and a jammi_grpc_refused_total{reason} counter increment (see Metrics below); see Configuration for the full [server.limits] reference and every default.

Lease and worker timing

One lease primitive ([lease], crates/jammi-db/src/catalog/lease.rs) owns every leased catalog row — a claimed job and a building result table alike — defaulting to a 30 s lease renewed every 10 s; the job worker adds a 1 s idle-poll ([worker]):

• Lease (30 s default) — how long a claimed job, or a result table being written, is exclusively owned by its holder before it becomes reclaimable.
• Heartbeat (10 s default) — renews the lease well inside the window, from the process’s shared lease-keeper thread: one registration per running job and one per BuildingTable between create_table and finish.
• Idle-poll (1 s default) — how often an idle worker checks for new work; reclaim runs on each idle tick, so a dead worker’s job is recovered within roughly one poll plus one lease. A dead writer’s building result table is reclaimed by the next session’s startup recovery sweep once its lease has expired — never before, so a replica restarting beside a live writer leaves that writer’s table alone.

The config layer enforces the invariant heartbeat × 2 < lease (and rejects a zero lease, zero heartbeat, or zero idle-poll). This guarantees a live holder renews at least twice per lease, so a single missed beat still leaves one in-window renewal that lands strictly before expiry — never coincident with it, which would race a reclaim. Bad values are rejected at config time, never silently clamped.

Job attempts cap

A job is retried at most 3 times. After the third attempt the expired-lease reclaim path fails the job for good rather than re-queueing it indefinitely.

GPU admission — a memory budget

GPU admission is a memory budget, not a max-concurrent-job count. The scheduler admits work against a budget of total_gpu_memory × (1 − headroom_fraction): a reservation is admitted by a compare-and-swap against the reserved total, and released via RAII when the permit drops. Many small jobs can run concurrently while one large job is admitted only when its memory fits the remaining budget.

Work queue

The work queue is the durable jobs table (migration 029) — one kind-agnostic table for every training AND compute kind, distinguishing a queued row a [worker] claim loop may pick up from an inline row a submitting call claims once, by id, in its own task. The queued path is drained with a SELECT … FOR UPDATE SKIP LOCKED claim (Postgres; SQLite’s single serialised writer) so concurrent workers each lock a distinct row. It is bounded by the lease plus the attempts cap, not by an in-memory buffer — there is no in-process queue-depth limit to overflow, and a worker crash leaves the row claimable again after the lease expires.

Close() and the successor handoff

Closing a session (InferenceSession::close / PyDatabase.close) shuts the process’s dedicated lease-keeper thread down and joins it — bounded by a 30s window — before closing the shared catalog pool. This ordering matters specifically for the SQLite backend: the keeper holds its own catalog connection on its own OS thread, independent of the shared pool, so closing only the shared pool while the keeper’s thread stays up leaves the unix-excl VFS’s process-scoped exclusive lock held and a successor process opening the same directory refused within its busy timeout, even after every other handle has let go. A keeper that does not exit within the shutdown window is logged and does not block the shared-pool close that follows — close() must never hang, even at the cost of leaving that one connection’s fate unresolved in the (unexpected) case the keeper’s thread is wedged.

Failure-mode matrix

reconcile boot cost, pre-layout keys, and delete_model

• reconcile_ready_manifests’s boot cost is one exists() per post-contract ready row (definition_hash IS NOT NULL) — it checks only that row’s .materialization.json sidecar is still present, never a full object-store LIST. A pre-contract row (definition_hash IS NULL) is skipped entirely; it legitimately has no sidecar to check.
• A key written before the tenant-prefixed layout landed is permanently unattributed. reconcile’s row→object and object→row checks both key off TenantSegment::parse, which only recognizes {seg}/{table}.parquet and models/{seg}/{job}/… shapes; an older flat key never round-trips through that parser, so it is reported as unattributed and never deleted at any grace — the allowlist is a safety property, not a migration path. Such an object stays reachable only by direct URL, exactly as it was before the tenant-prefixed layout existed. unattributed is reported only by the admin cross-tenant pass (reconcile --all); a tenant-scoped pass never lists another tenant’s — or nobody’s — stray keys.
• reconcile’s grace gate compares two DIFFERENT clocks. The age check (is an orphan candidate old enough to reclaim?) reads last_modified off the OBJECT STORE’s own clock and compares it against the replica running reconcile’s Utc::now() — never the catalog database’s clock the way a lease predicate does. apply=true requiring grace >= the configured lease duration is exact only when the object store’s clock and the replica’s clock agree; ordinary NTP-level skew erodes the safety margin, it does not remove the mechanism — set grace to several multiples of the lease duration as the practical guard against both a slow writer and clock skew together.
• delete_model deletes the catalog row only — it never touches the artifact bytes at models/{seg}/{job_id}/…, and it refuses (JammiError::ModelReferenced) while another row still references it. A specific {model_id}:epoch_N checkpoint row is independent of the model’s own row: deleting one leaves the other’s row and bytes untouched. An artifact prefix no models row names any longer becomes a reconcile orphan candidate on the next pass, aged against grace like any other.

Catalog durability under crash vs. power loss

Result-table crash-consistency reconciles whatever the catalog durably retained against the bytes on disk, so the catalog’s own durability setting bounds the guarantee:

• Process crash (the engine dies, the host survives): both backends replay their write-ahead log on restart, so a building → ready (or the building insert recovery later reconciles) that committed before the crash is present after it. No committed catalog state is lost.
• Host power loss: Postgres commits synchronously (fsync per commit by default), so a committed transaction survives. SQLite runs synchronous=NORMAL under WAL — it fsyncs at checkpoint, not on every commit — so a power loss can lose the last committed transaction(s) since the previous checkpoint. A row that was not durably retained simply isn’t seen by recovery; the bytes it would have pointed at are reaped as an orphan on a later sweep. This is a property of the catalog backend’s durability configuration, not of the reconciliation.

The trigger stream’s at-least-once guarantee inherits exactly this bound, because its durable log is the same kind of backing table written through the same backend transaction. Under a process crash the at-least-once guarantee is unconditional — the committed backing-table rows replay on reconnect. Under a host power loss the durable log itself is power-loss-bounded: on Postgres a committed publish survives; on SQLite (synchronous=NORMAL under WAL) the last committed backing-table row(s) since the previous checkpoint can be lost, and an offset whose row was not durably retained will not replay. At-least-once is therefore full on Postgres and power-loss-bounded on SQLite — never weaker than the backing log’s own durability.