Operability

How to run a Jammi server in production: what it exposes for observability, how it shuts down cleanly, the resource limits it enforces, and how it behaves when a dependency fails. Everything below describes the system as it ships today.

Observability surface

The server exposes three HTTP side-channel endpoints, independent of the gRPC and Flight SQL data paths:

/healthz answers without touching any dependency, so an orchestrator uses it to decide whether to restart the container. /readyz goes one step further and pings the catalog — a transient catalog outage returns 503 so the load balancer removes the instance from rotation rather than restarting it.

$ curl -s localhost:8080/healthz
{"status":"ok","version":"0.29.0"}

$ curl -s localhost:8080/readyz          # catalog reachable
{"status":"ready"}

$ curl -s localhost:8080/readyz          # catalog unreachable → 503
{"status":"not_ready","detail":"catalog ping failed: connection refused"}

Metrics

/metrics emits four substrate-level metrics that the gRPC services and Flight SQL layer feed:

# HELP jammi_grpc_requests_total Total number of gRPC requests served across all jammi.v1 services.
# TYPE jammi_grpc_requests_total counter
jammi_grpc_requests_total 1432
# HELP jammi_search_latency_seconds Vector-search request latency, in seconds.
# TYPE jammi_search_latency_seconds histogram
jammi_search_latency_seconds_bucket{le="0.05"} 311
jammi_search_latency_seconds_bucket{le="0.1"} 402
jammi_search_latency_seconds_sum 27.41
jammi_search_latency_seconds_count 418

Tracing

The server installs a global tracing subscriber. Spans carry the correlation fields that let you follow a request across the gRPC surface and the worker fleet:

• gRPC handler spans carry tenant_id — recorded once the handler has resolved the request’s tenant scope.
• run_claimed_job (the worker dispatching a claimed job) carries worker_id, job_id, and tenant_id.
• run_spec (the training run inside a claimed job) carries job_id and worker_id.

Logs are emitted as structured records, JSON or human-readable text per logging.format (LogFormat). The filter comes from logging.level, with RUST_LOG as an optional override. Output always goes to stdout — a server runs non-interactively by design — and ANSI colour is enabled only when stdout is a terminal.

{"timestamp":"2026-06-15T04:17:33.114Z","level":"INFO","fields":{"message":"job completed"},"target":"jammi_ai::fine_tune::worker","span":{"job_id":"job-7af3","worker_id":"worker-2","tenant_id":"acme","name":"run_claimed_job"}}

Graceful shutdown

run_with_shutdown drives both the HTTP side-channel and the gRPC surface and drains them in parallel: the call returns once both have stopped accepting new connections and finished serving in-flight requests. The standalone binary wires both SIGINT (Ctrl+C) and SIGTERM, so docker stop — which sends SIGTERM — triggers the same clean drain as an interactive Ctrl+C.

Backpressure and resource limits

The engine enforces these limits. They are the only ones it enforces — there is no configured gRPC message-size cap and no in-memory worker-queue-depth bound; the work queue is durable, not buffered (see below).

Worker timing

The training worker drives its loop on three intervals, defaulting to 30 s lease / 10 s heartbeat / 1 s idle-poll:

• Lease (30 s default) — how long a claimed job is exclusively owned before it becomes reclaimable.
• Heartbeat (10 s default) — renews the lease well inside the window.
• Idle-poll (1 s default) — how often an idle worker checks for new work; reclaim runs on each idle tick, so a dead worker’s job is recovered within roughly one poll plus one lease.

The config layer enforces the invariant heartbeat × 2 < lease (and rejects a zero heartbeat or zero idle-poll). This guarantees a live worker renews at least twice per lease, so a single missed beat still leaves one in-window renewal that lands strictly before expiry — never coincident with it, which would race an idle-polling worker’s reclaim. Bad values are rejected at config time, never silently clamped.

Job attempts cap

A job is retried at most 3 times. After the third attempt the expired-lease reclaim path fails the job for good rather than re-queueing it indefinitely.

GPU admission — a memory budget

GPU admission is a memory budget, not a max-concurrent-job count. The scheduler admits work against a budget of total_gpu_memory × (1 − headroom_fraction): a reservation is admitted by a compare-and-swap against the reserved total, and released via RAII when the permit drops. Many small jobs can run concurrently while one large job is admitted only when its memory fits the remaining budget.

Work queue

The work queue is the durable Postgres training_jobs table, drained with a SELECT … FOR UPDATE SKIP LOCKED claim so concurrent workers each lock a distinct row. It is bounded by the lease plus the attempts cap, not by an in-memory buffer — there is no in-process queue-depth limit to overflow, and a worker crash leaves the row claimable again after the lease expires.

Failure-mode matrix

Catalog durability under crash vs. power loss

Result-table crash-consistency reconciles whatever the catalog durably retained against the bytes on disk, so the catalog’s own durability setting bounds the guarantee:

• Process crash (the engine dies, the host survives): both backends replay their write-ahead log on restart, so a building → ready (or the building insert recovery later reconciles) that committed before the crash is present after it. No committed catalog state is lost.
• Host power loss: Postgres commits synchronously (fsync per commit by default), so a committed transaction survives. SQLite runs synchronous=NORMAL under WAL — it fsyncs at checkpoint, not on every commit — so a power loss can lose the last committed transaction(s) since the previous checkpoint. A row that was not durably retained simply isn’t seen by recovery; the bytes it would have pointed at are reaped as an orphan on a later sweep. This is a property of the catalog backend’s durability configuration, not of the reconciliation.

The trigger stream’s at-least-once guarantee inherits exactly this bound, because its durable log is the same kind of backing table written through the same backend transaction. Under a process crash the at-least-once guarantee is unconditional — the committed backing-table rows replay on reconnect. Under a host power loss the durable log itself is power-loss-bounded: on Postgres a committed publish survives; on SQLite (synchronous=NORMAL under WAL) the last committed backing-table row(s) since the previous checkpoint can be lost, and an offset whose row was not durably retained will not replay. At-least-once is therefore full on Postgres and power-loss-bounded on SQLite — never weaker than the backing log’s own durability.