Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Reference Topologies

The same engine binary — and the same Rust crate / Python wheel for the embedded case — serves every deployment shape below. Nothing here is a different code path: each shape is a point on the backend-driver configuration surface (catalog, trigger broker, object storage) plus a process count. This page pins each shape to the concrete artifact that realises it: a config, a tested deploy/ Compose file, or a kustomize manifest tree.

Shape A — single-process embedded

Artifact: the jammi-db / jammi-ai Rust crates, or the jammi-ai Python wheel. No server process.

The workspace defaults ARE Shape A: SQLite catalog under artifact_dir, result tables on local disk, the in-memory trigger broker, an in-process model cache. No config file is required.

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
use jammi_db::config::JammiConfig;
use jammi_ai::session::InferenceSession;

async fn run() -> Result<(), Box<dyn std::error::Error>> {
let config = JammiConfig::load(None)?; // SQLite + local fs + in-memory broker
let session = InferenceSession::new(config).await?;
let _ = session;
Ok(())
}
}
import jammi

# `file://` resolves to the compiled in-process engine (the `jammi-ai[embedded]`
# extra), constructed through the same `JammiConfig` resolution chain as
# `jammi-server` — via the engine's own `jammi_native.open_local`.
db = jammi.connect("file:///var/lib/jammi")

Notebooks, the jammi CLI against a local directory, single-machine batch jobs, and laptop development all run this shape unmodified — see Quickstart: Rust and Quickstart: Python.

Shape B — single-tenant server

Artifact: deploy/docker-compose.yml — one jammi-server process, a Postgres catalog, and a JetStream broker, tested end to end by the compose-smoke workflow on every push to main and nightly.

# Shape B: `jammi-server` against Postgres (catalog) + JetStream (broker) —
# a worked example of a *configuration* change, not orchestration policy
# (orchestration is the deployer's runtime; see `docs/guide/src/philosophy.md`).
# Every knob below is one this compose file only sets, never invents: the
# image's env-only configuration layer (`crates/jammi-db/src/config/`)
# resolves `JAMMI_CATALOG__POSTGRES__URL`, `JAMMI_BROKER__JET_STREAM__URL`,
# and `JAMMI_SERVER__SERVICES` exactly as documented in
# `docs/guide/src/reference-topologies.md`.
#
# `jammi-server`'s healthcheck is exec-form `jammi-server probe` — the
# runtime image is distroless (no shell, no curl/wget), so the readiness
# probe is a generic CLI subcommand the binary itself ships (see
# `crates/jammi-server/src/probe.rs`), the same shape as Postgres's own
# `pg_isready`.
#
# `secrets` live in a git-ignored `.env` file next to this compose file;
# `deploy/.env.example` documents the one required key.
#
# This file has no compute-tier service: Kubernetes runs the compute tier as
# a `StatefulSet` (`deploy/kubernetes/overlays/shape-d`) because a rank's
# peer address must survive a pod restart; Compose has no ordinal-stable
# identity primitive, so a multi-rank Compose deployment lists each rank as
# its own named service instead (`deploy/kubernetes/README.md`).

services:
  jammi-server:
    image: ${JAMMI_IMAGE:-ghcr.io/f-inverse/jammi-ai-server:latest}
    # For local developers only: `docker compose build` compiles this repo's
    # own Dockerfile (CPU variant) instead of pulling the published image.
    # The CI smoke override (`docker-compose.ci.yml`) drops this block and
    # points `JAMMI_IMAGE` at the image it already built + loaded.
    build:
      context: ..
      dockerfile: Dockerfile
    environment:
      JAMMI_CATALOG__POSTGRES__URL: postgres://jammi:jammi@postgres:5432/jammi
      JAMMI_BROKER__JET_STREAM__URL: nats://nats:4222
      JAMMI_SERVER__SERVICES: all
      JAMMI_AUDIT_MASTER_KEY: ${JAMMI_AUDIT_MASTER_KEY:?set it in .env}
    ports:
      # Loopback-bound: the deployer publishes these for a
      # TLS-terminating proxy running on the same host to reach (see
      # `docs/guide/src/security.md`'s transport-encryption section), never
      # for direct exposure to an untrusted network. TLS termination is the
      # deployer's runtime, not this compose file's job.
      - "127.0.0.1:8081:8081" # gRPC + Flight SQL
      - "127.0.0.1:8080:8080" # HTTP side-channel (/healthz, /readyz, /metrics)
    volumes:
      # Artifacts (indices, materialized Parquet, `index_cache`) and the
      # Hugging Face Hub cache (`HF_HOME`, set by the image) — a named
      # volume so docker provisions ownership for distroless's nonroot
      # uid (65532); a bind mount would need host-side `chown` first.
      - jammi-data:/var/lib/jammi
    depends_on:
      postgres:
        condition: service_healthy
      nats:
        condition: service_healthy
    restart: unless-stopped
    # `docker stop` / `docker compose down` send SIGTERM (DRAIN: the in-flight
    # job finishes) and SIGKILL after this grace; 120 s covers a small epoch.
    # SIGINT is RELEASE — the job's lease is handed back at once and the next
    # process to claim it resumes it. `docker compose kill -s SIGINT` delivers
    # that signal but the daemon records a kill as a MANUAL stop, and a manually
    # stopped container is exempt from `restart: unless-stopped`: nothing comes
    # back on its own. To release AND have this policy restart the container,
    # signal the process from the host (`kill -INT $(docker inspect -f
    # '{{.State.Pid}}' <container>)`, what tests/compose/shape_b_release.py
    # does), or follow a compose kill with `docker compose start jammi-server`.
    stop_grace_period: 120s
    healthcheck:
      test: ["CMD", "/usr/local/bin/jammi-server", "probe"]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 10s

  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: jammi
      POSTGRES_PASSWORD: jammi
      POSTGRES_DB: jammi
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "jammi"]
      interval: 5s
      timeout: 5s
      retries: 12

  nats:
    image: nats:2.10-alpine
    command: ["-js", "-m", "8222"]
    healthcheck:
      # The image ships no shell — `wget` is the one HTTP-capable binary
      # nats:2.10-alpine's base carries; `nats-server --help` (no shell
      # dependency either) would prove only that the binary exists, not
      # that JetStream came up.
      test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:8222/healthz"]
      interval: 5s
      timeout: 5s
      retries: 12

volumes:
  jammi-data:
  postgres-data:

The jammi-server service’s configuration is entirely environment-driven, through the same JAMMI_<PATH> layer every deployment shape uses:

  • JAMMI_CATALOG__POSTGRES__URL — the Postgres catalog connection.
  • JAMMI_BROKER__JET_STREAM__URL — the JetStream broker connection.
  • JAMMI_SERVER__SERVICES — all here (every compiled-in service tier); see Service tiers for narrower selections.
  • JAMMI_AUDIT_MASTER_KEY — read from deploy/.env (copy deploy/.env.example); an absent key makes Compose itself refuse to come up (${JAMMI_AUDIT_MASTER_KEY:?…}) before the container ever starts. Signing is not a “disabled” mode: it stays configured and, unset, only fails at the first audit write; a present-but-malformed key instead makes jammi-server refuse to start.

The published ports (8081, 8080) are bound to 127.0.0.1, not 0.0.0.0: the compose file publishes them for a TLS-terminating proxy running on the same host to reach, never for direct exposure to an untrusted network (see Security Posture).

The healthcheck is exec-form jammi-server probe (see jammi-server probe below) — the runtime image is distroless and ships no shell, so a curl/wget-based HEALTHCHECK is not an option; probe is the same generic-CLI shape as Postgres’s own pg_isready.

What the smoke proves. tests/compose/remote_smoke.py is the shared smoke oracle; shape_b_remote.py (this Compose shape) and shape_c_kube_remote.py (the Kubernetes shape below) are its two drivers, differing only in how they restart the server and in what they assert afterwards — durability on the Compose volume here, the shared catalog on the emptyDir pods there. The oracle asserts get_server_info().broker == "jet_stream" — the RUNTIME driver kind, which fails if JAMMI_BROKER__JET_STREAM__URL were ever dropped from the compose file, unlike the compile-time features list alone — registers the bundled patents.parquet fixture, generates embeddings with the bundled tiny_bert fixture, searches for the stored vector’s own nearest neighbor (an exact self-hit), then restarts the jammi-server container and repeats the same search. It asserts durability across the restart: identical result ids and scores within 1e-6, and an unchanged list_index_segments — the segment bundle on the Postgres-backed catalog and the jammi-data volume survives a container restart, not merely that the server answers again afterward. The workflow itself queries the postgres container directly afterward for the sources table’s row count — the matching runtime oracle on the catalog side, failing if JAMMI_CATALOG__POSTGRES__URL were ever dropped.

Shape C — multi-tenant server

Artifact: N jammi-server replicas behind a load balancer, a shared Postgres catalog, a shared object store, and a shared JetStream (or Postgres-as-)broker. Every replica runs the identical config; only the process count differs from Shape B.

artifact_dir = "/var/lib/jammi"

[catalog.postgres]
url = "${POSTGRES_URL}?sslmode=verify-full&sslrootcert=/etc/ssl/certs/ca-certificates.crt"
pool_size = 16
max_lifetime_secs = 1800

[broker.jet_stream]
url = "nats://${NATS_HOST}:4222"
retention_seconds = 604800
credentials = { file = "/var/run/secrets/nats.creds" }

[lease]
duration_secs = 30
heartbeat_secs = 10

[server]
services = ["core", "event", "eval"]

[worker]
enabled = false

The env-only equivalent (see Deploy as a Server: a production shape, entirely from the environment for the fuller walkthrough, including the object-store result root and a file-backed audit signing key):

export JAMMI_CATALOG__POSTGRES__URL="postgres://jammi:${POSTGRES_PASSWORD}@postgres.internal:5432/jammi?sslmode=verify-full&sslrootcert=/etc/ssl/certs/ca-certificates.crt"
export JAMMI_CATALOG__POSTGRES__POOL_SIZE=16
export JAMMI_BROKER__JET_STREAM__URL="nats://nats.internal:4222"
export JAMMI_BROKER__JET_STREAM__CREDENTIALS__FILE=/run/secrets/nats.creds
export JAMMI_LEASE__DURATION_SECS=30
export JAMMI_LEASE__HEARTBEAT_SECS=10
export JAMMI_SERVER__SERVICES=all
jammi-server

The guarantee. With every replica pointed at the same Postgres catalog, the one lease primitive documented in Catalog Backend and Trigger Broker: Multi-writer safety holds across the whole fleet: concurrent writers never corrupt or reap each other’s building tables; a crashed replica’s rows are reclaimed at the next session boot or by jammi reconcile; migrations are serialised by an advisory lock. [lease] (duration_secs / heartbeat_secs, JAMMI_LEASE__*) is the one timing knob every leased catalog row shares across the fleet — see Configuration for its full field reference.

Kubernetes (deploy/kubernetes)

Orchestration — which scheduler, how replicas are placed, ingress, TLS termination, autoscaling — is the deployer’s runtime, not the engine’s (see “How it deploys”). deploy/kubernetes/base is the whole shape the engine cares about — a Deployment with readinessProbe against /readyz and runAsNonRoot, a Service, a ConfigMap of the non-secret knobs, an emptyDir for scratch; ingress, TLS, autoscaling, network policy and each cloud’s managed-service annotations are the deployer’s overlay, and the seam is a kustomize patch on the Deployment/Service metadata, never a change to the engine’s knobs. Every PR validates kustomize build + kubeconform --strict --kubernetes-version 1.34.11 over every kustomization in the tree, in ci.yml’s Guard (kubernetes manifests); the kube-smoke workflow additionally stands the ci overlay up on a real kind cluster on push to main, nightly, on manual dispatch, and on any pull request that touches deploy/kubernetes/** or tests/compose/**.

# Shape C's query tier -- the artifact the guide's Kubernetes section
# (docs/guide/src/reference-topologies.md) includes directly. See
# `deploy/kubernetes/README.md` for the boundary this file does and does not
# cover.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jammi-server
spec:
  # Every replica runs the identical config against the same Postgres catalog
  # and JetStream broker; the one lease + advisory-lock mechanism
  # (docs/guide/src/reference-topologies.md#shape-c--multi-tenant-server,
  # docs/guide/src/catalog-and-broker.md#multi-writer-safety; #479 closed)
  # is what makes concurrent replicas safe. Raise or lower freely.
  replicas: 3
  selector:
    matchLabels: { app: jammi-server }
  template:
    metadata:
      labels: { app: jammi-server }
    spec:
      # Termination grace is the Kubernetes default (30 s): SIGTERM drains —
      # in-flight requests finish, an idle WaitJob/Subscribe stream ends with
      # UNAVAILABLE "server draining" — then SIGKILL. A training job still in
      # flight past the grace resumes from its last epoch bundle after one
      # lease window and consumes one attempt; raise the grace to the compute
      # overlay's value (overlays/shape-d) when this base runs training in
      # earnest.
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        # The scratch emptyDir below must be writable by uid 65532 without
        # relying on kubelet's default 0777 volume permissions.
        fsGroup: 65532
      containers:
        - name: jammi-server
          # Pin-advice: `:latest` is re-pointed by every `v*` release tag.
          # Pin an exact `:vX.Y.Z` tag for reproducible deploys.
          image: ghcr.io/f-inverse/jammi-ai-server:latest
          ports:
            - { containerPort: 8081, name: flight }
            - { containerPort: 8080, name: health }
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          envFrom:
            # JAMMI_AUDIT_MASTER_KEY, JAMMI_CATALOG__POSTGRES__URL,
            # JAMMI_BROKER__JET_STREAM__URL -- no Secret manifest ships in
            # git; create it out-of-band (see the README).
            - secretRef: { name: jammi-server-secrets, optional: false }
          volumeMounts:
            - { name: config, mountPath: /etc/jammi, readOnly: true }
            - { name: scratch, mountPath: /var/lib/jammi }
      volumes:
        - name: config
          configMap: { name: jammi-server-config } # the non-secret knobs only
        - name: scratch
          emptyDir: {} # local scratch only -- the catalog/broker/result-root carry the durable state

apiVersion: v1
kind: Service
metadata:
  name: jammi-server
spec:
  # No `type:` -> ClusterIP (the default): cluster-internal only, reachable
  # from other pods/Services on this cluster, not from outside it -- pair
  # with an Ingress/Gateway and a TLS terminator to reach it externally.
  selector: { app: jammi-server }
  ports:
    - { name: flight, port: 8081, targetPort: 8081 }
    - { name: health, port: 8080, targetPort: 8080 }

# env wins over this file (the JAMMI_<PATH> layer resolves after it); secrets
# never here -- JAMMI_AUDIT_MASTER_KEY, JAMMI_CATALOG__POSTGRES__URL and
# JAMMI_BROKER__JET_STREAM__URL come from the Secret this Deployment's
# envFrom names, never from this ConfigMap.

[storage]
result_root = "s3://jammi-results/prod"

[storage.cloud.s3]
region = "us-east-1"

[lease]
duration_secs = 30
heartbeat_secs = 10

[distributed]
# Read at the submit edge only (this query tier is where jobs are
# enqueued; a compute pod never reads it): admits a `world_size` up to the
# compute pod's two devices. A cross-pod `Peer` gang needs this bound
# raised to the target world size here, and at least that many compute
# pods able to hold a rank.
max_world_size = 2

[server]
services = ["core", "event", "eval"]

[worker]
enabled = false

jammi-server-secrets is a Secret carrying the same env keys the Compose and bare-env forms above use — JAMMI_CATALOG__POSTGRES__URL, JAMMI_BROKER__JET_STREAM__CREDENTIALS__FILE (or __URL), JAMMI_AUDIT_MASTER_KEY — mounted as env vars, never baked into the ConfigMap. No Secret manifest ships in git; create it out-of-band, once per cluster namespace:

kubectl -n <namespace> create secret generic jammi-server-secrets \
  --from-literal=JAMMI_AUDIT_MASTER_KEY=<...> \
  --from-literal=JAMMI_CATALOG__POSTGRES__URL=<...> \
  --from-literal=JAMMI_BROKER__JET_STREAM__URL=<...>

never in git.

What kube-smoke proves. Against the ci overlay on a real kind cluster: readiness (kubectl rollout status), get_server_info().broker == "jet_stream", the Postgres sources table’s row count via psql against the postgres StatefulSet, one-hop image identity — every pod’s containerStatuses[].imageID traces back to the image kind loaded, never a registry pull — and a rollout restart after which the new pod still sees the source the pod it replaced registered. What it does NOT prove: durability across that restart — the scratch volume is an emptyDir, so the Postgres catalog and the JetStream broker, not the pod’s local disk, carry the state a fresh pod recovers.

Shape D — disaggregated

Artifact: the query tier above (Shape C’s Deployment) unchanged, plus a GPU-scheduled StatefulSet compute tier and a single-replica CPU scheduler Deployment, both running the SAME image family, scheduled separately.

Running jobs is not a service tier (see Service tiers): whether a process claims and executes the jobs it accepted is [worker] enabled. Every query-tier replica runs [worker] enabled = false (JAMMI_WORKER__ENABLED=false) — it still mounts core/event/eval and accepts every submission — and both compute-tier roles below run [worker] enabled = true (JAMMI_WORKER__ENABLED=true) so only they run the job worker’s claim loop against the shared catalog. Their kinds differ: the compute StatefulSet’s pods claim ["fine_tune", "graph_fine_tune", "context_predictor"], the scheduler Deployment claims ["fine_tune", "graph_fine_tune"] only — context_predictor has no placed arm, so listing it on the CPU scheduler pod would train it there instead of on a device. [server] services = [] on both — a pure compute/scheduler node serves no query-tier gRPC.

Two compute-tier roles, one config knob. Whether a process hosts a Ballista scheduler or executor (or neither) is [ballista] (scheduler_bind / executor, see Configuration) — a process with neither role runs exactly as it always has:

  • The scheduler is ONE dedicated single-replica Deployment (jammi-server-scheduler): [ballista] scheduler_bind set, no [ballista.executor], CPU image. It claims a training job and PLACES it — as one Ballista task — on a registered compute-pod executor; when no executor is registered yet it claims and runs the job in-process instead (byte-identical either way, per device kind), since it is also a plain worker-enabled fleet member. A third arm: when a live registered executor exists but none of its own devices lists the plan’s device kind (a row a dead executor left behind is not live and never counts), the submission is refused typed BEFORE it ever reaches the scheduler — the row is left running for reclaim (an attempt spent), never run in-process on the claiming pod.
  • The compute tier’s StatefulSet pods (jammi-server-compute) each host a Ballista EXECUTOR ([ballista.executor] pointed at the scheduler’s Service) alongside their own [worker] enabled = true claim loop: a pod claims and runs a job in-process exactly like the scheduler can, or accepts a gang the scheduler placed on it. GangExec’s world is informational only — topology is decided on the pod that actually runs the body, from its OWN [worker] local_ranks, never from the submitter’s.

Placement is decided BEFORE topology, for every fine_tune/ graph_fine_tune attempt a scheduler-role process claims (any process whose HostAdmission exposes a placement submitter): it always attempts to place the whole job as one Ballista task on a registered compute-pod executor, regardless of W versus [worker] local_ranks. Only the pod that ends up running the job’s coordinator body — the placed executor, or the claiming process itself when no submitter seam exists or no other executor is registered — decides Single/Local/Peer from its OWN local_ranks: this overlay admits single-pod gangs (W ≤ 2, Local on one pod’s two devices); a cross-pod Peer gang of world W needs W > local_ranks, max_world_size ≥ W on the submit edge (base/jammi.toml — the key is read only where jobs are enqueued), and at least W compute pods able to hold a rank (the coordinator’s included) — the submitting host moves to an Awaiting holder state for the whole placement (it runs no compute meanwhile, but can still serve a RunRank session). Placement always excludes a task’s own submitter: a claimant’s host is never bound its own gang, so a lone compute pod placing its own claim would deadlock against itself — this is why the scheduler is a SEPARATE role rather than “whichever compute pod claims first places its own siblings.”

Each jammi-server-compute pod’s peer_advertise is its own stable DNS name under the headless Service (<pod>.jammi-server-compute.<namespace>.svc.cluster.local, publishNotReadyAddresses: true so a rank can dial a sibling that is still warming), the property a plain Deployment’s churning pod names cannot hold; its Ballista advertise_host is the SAME per-pod name, since the scheduler must dial the executor back on the identical stable address.

Both compute-tier roles carry terminationGracePeriodSeconds: 600 — SIGTERM drains (the in-flight training job finishes, every epoch bundle lands) and SIGKILL follows the grace; SIGINT, or jammi-server release from a preStop hook, RELEASES — on a CONFIRMED release (exit 0), the job’s lease is handed back at once and any other replica claims it within one idle poll at no attempt cost; a DEGRADED release (exit 3) does not cost an attempt universally, and its per-lease outcome depends on which determinant degraded (see the RELEASE breakdown in deploy/kubernetes/README.md and deploy-server.md below — never assume the CONFIRMED cost here for a degraded exit). The grace must cover one epoch’s wall time; on spot capacity use RELEASE. DRAIN on an executor pod additionally stops Ballista task admission at once — the executor reports Terminating to the scheduler the instant DRAIN begins, before the in-flight worker job is joined, and a terminating executor is never bound; a gang the pod is still dialled with inside its grace is refused before any claim transfer — but waits for any in-flight placed gang before the process itself stops — only RELEASE tears the executor down immediately. The operative rule, the rollout arithmetic and both preStop recipes are in deploy/kubernetes/README.md (“Shutdown: DRAIN and RELEASE”); the modes themselves are in Shutdown.

The compute tier is a StatefulSet — each rank’s peer address (and Ballista advertise_host) must stay stable across a pod restart, which a Deployment’s churning pod names cannot hold. This overlay is validated by kubeconform only — CI has no GPU node.

# jammi-server-compute ships as a StatefulSet, not a Deployment: each rank's
# peer address must stay stable across a pod restart for `instances.peer_addr`
# gang membership (`[server] peer_advertise`, DESIGN.md §7) -- a Deployment's
# pod names churn on every reschedule and cannot serve as that identity. The
# headless Service (service-compute-headless.yaml) is the DNS half of the
# same fact: ranks dial each other directly on the peer listener
# (`peer_advertise`), never through a load balancer or a virtual IP.
#
# `podManagementPolicy: Parallel`: ranks assemble by catalog membership
# (`instances` rows, DESIGN.md §7), never by ordinal start order, so no rank
# waits on a lower-ordinal sibling to become Ready before it starts.
#
# `nvidia.com/gpu: 2` matches `[worker] local_ranks = 2` in
# jammi-compute.toml: one device per local rank (`[gpu] devices = [0, 1]`).
#
# Each pod ALSO registers as a Ballista executor with the scheduler
# Deployment (deployment-scheduler.yaml): `ballista-flt`/`ballista-grpc`
# below are the shuffle/task listeners `[ballista.executor]`
# (jammi-compute.toml) binds, advertised at this same pod's stable DNS name
# (`JAMMI_BALLISTA__EXECUTOR__ADVERTISE_HOST` below) -- a training job
# claimed by the scheduler pod is PLACED onto whichever of these pods the
# scheduler's device-aware policy picks; a training job claimed by one of
# these pods itself runs here in-process.
#
# kubeconform-validated only -- no GPU node is available in CI, so this
# StatefulSet never runs a real pod in the kind smoke.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jammi-server-compute
spec:
  serviceName: jammi-server-compute
  replicas: 2
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels: { app: jammi-server-compute }
  template:
    metadata:
      labels: { app: jammi-server-compute }
    spec:
      # DRAIN on SIGTERM finishes the in-flight job; RELEASE (SIGINT /
      # `jammi-server release` via preStop) hands it to a successor at once.
      # 600 s = cluster-autoscaler `--max-graceful-termination-sec` default; a
      # larger value is honoured only by rollouts/`kubectl delete` — raise both
      # together. The operative rule and the rollout arithmetic are in the
      # README ("Shutdown: DRAIN and RELEASE").
      terminationGracePeriodSeconds: 600
      nodeSelector:
        gpu-node-pool: "true" # your cluster's own GPU node label
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532
      containers:
        - name: jammi-server
          # amd64 only. Pin-advice: `:latest` is re-pointed by every `v*`
          # release tag; pin an exact `:vX.Y.Z` tag for reproducible
          # GPU-node deploys.
          image: ghcr.io/f-inverse/jammi-ai-server-cu12:latest
          resources:
            limits: { nvidia.com/gpu: 2 }
          ports:
            - { containerPort: 8081, name: flight }
            - { containerPort: 8080, name: health }
            - { containerPort: 9000, name: peer }
            - { containerPort: 50051, name: ballista-flt }
            - { containerPort: 50052, name: ballista-grpc }
          env:
            # The downward API's pod-field env source
            # (EnvVarSource.fieldRef; `metadata.name` / `metadata.namespace`
            # resolve to THIS pod's own values, not a template string).
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - name: POD_NAMESPACE
              valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
            # Kubernetes expands `$(VAR)` in an `env[].value` string only for
            # a variable name declared EARLIER in this same list (the
            # `EnvVar.value` field doc: "Variable references $(VAR_NAME) are
            # expanded using the previously defined environment variables in
            # the container and any service environment variables" --
            # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.34/#envvar-v1-core)
            # -- POD_NAME/POD_NAMESPACE above must stay ordered before this
            # key. This is the pod's stable identity for gang membership: the
            # ordinal-indexed DNS name the headless Service publishes
            # (`<pod>.jammi-server-compute.<namespace>.svc.cluster.local`),
            # never a virtual IP. `[server] peer_bind` (the listen side, the
            # same on every replica) lives in jammi-compute.toml instead --
            # one non-secret knob file for the shared value, env only for the
            # per-pod identity that file cannot express.
            - name: JAMMI_SERVER__PEER_ADVERTISE
              value: "$(POD_NAME).jammi-server-compute.$(POD_NAMESPACE).svc.cluster.local:9000"
            # The SAME per-pod stable DNS name, host only (no port -- this
            # key is `[ballista.executor] advertise_host`, jammi-compute.toml,
            # a bare host the scheduler dials back on the flight/grpc ports
            # THAT config table already names).
            - name: JAMMI_BALLISTA__EXECUTOR__ADVERTISE_HOST
              value: "$(POD_NAME).jammi-server-compute.$(POD_NAMESPACE).svc.cluster.local"
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          envFrom:
            # Same catalog/broker Secret as the query tier.
            - secretRef: { name: jammi-server-secrets, optional: false }
          volumeMounts:
            - { name: config, mountPath: /etc/jammi, readOnly: true }
            - { name: scratch, mountPath: /var/lib/jammi }
      volumes:
        - name: config
          configMap: { name: jammi-compute-config }
        - name: scratch
          # The cu12 image also keeps CUDA_CACHE_PATH under /var/lib/jammi
          # alongside HF_HOME -- local scratch only, no durable state here.
          emptyDir: {}

# Headless (clusterIP: None): ranks resolve each other by pod DNS name
# (`<pod>.jammi-server-compute.<namespace>.svc.cluster.local`, the value
# `[server] peer_advertise` is set to per-pod in statefulset-compute.yaml's
# `env`) and dial that name directly on the peer port -- never a virtual IP,
# never through this Service acting as a load balancer.
#
# `publishNotReadyAddresses: true`: the peer/gang listener is bound and
# already serving `PeerService`/`GangService` connections as soon as
# `OssServer::bind` returns (`crates/jammi-server/src/runtime.rs`), which
# happens BEFORE the process's own `/readyz` reports ready --
# `/readyz` gates the catalog ping and, when `[server] preload_models` is
# set, model preload (`docs/guide/src/deploy-server.md`'s "Preloading
# models": the claim loop "waits at its gate with its `workers.state` row
# reading `warming`" until then). A coordinator only dials a peer AFTER it
# claims a job, i.e. after its OWN `/readyz` is ready, but the PEER it
# dials may still be warming. Without this flag a not-yet-ready pod has no
# DNS record on a headless Service (the default EndpointSlice/DNS behaviour
# publishes only Ready pod addresses), so the coordinator would get NXDOMAIN
# resolving a peer whose `GangService` is already live and reachable.
apiVersion: v1
kind: Service
metadata:
  name: jammi-server-compute
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector: { app: jammi-server-compute }
  ports:
    - { name: flight, port: 8081, targetPort: 8081 }
    - { name: health, port: 8080, targetPort: 8080 }
    - { name: peer, port: 9000, targetPort: 9000 }
    - { name: ballista-flt, port: 50051, targetPort: 50051 }
    - { name: ballista-grpc, port: 50052, targetPort: 50052 }

# env wins over this file (the JAMMI_<PATH> layer resolves after it); secrets
# never here -- see base/jammi.toml's header.

[storage]
result_root = "s3://jammi-results/prod"

[storage.cloud.s3]
region = "us-east-1"

[lease]
duration_secs = 30
heartbeat_secs = 10

[gpu]
# The ordered device list this pod's ranks place on: rank i on devices[i].
# devices[0] must equal `device` (unset here, default 0), so no separate
# `device` key is needed. Two entries for the shape's two local ranks below.
devices = [0, 1]

# This pod also hosts a Ballista EXECUTOR -- it registers with the
# scheduler Deployment's Service and accepts a PLACED training job (one
# `GangExec` task) alongside the jobs it claims and runs on its own,
# in-process. One task slot: a placed gang and this pod's own in-process
# claim both take the SAME `HostAdmission` job slot (this pod coordinates
# one placed task at a time), so a second bound task is refused typed at
# the slot rather than left to contend for `[gpu] devices` capacity.
[ballista.executor]
scheduler_address = "jammi-server-scheduler:50050"
bind = "0.0.0.0:50051"
grpc_bind = "0.0.0.0:50052"
# The per-pod stable DNS name (statefulset-compute.yaml's downward-API
# `JAMMI_BALLISTA__EXECUTOR__ADVERTISE_HOST`, same construction as
# `peer_advertise` below) -- never set here, since every replica shares this
# one ConfigMap and the value differs per pod.
task_slots = 1

# Pure compute node: mounts no query-tier gRPC service, only claims and runs
# queued jobs (in-process or as a placed Ballista task).
[server]
services = []
# This pod's own peer/gang listener, identical on every replica of this
# StatefulSet: "0.0.0.0:PORT" accepts a connection on any interface. The
# per-pod dial-in address other replicas use is `peer_advertise`, set from
# the downward API in statefulset-compute.yaml's `env` -- never here, since
# every replica shares this one ConfigMap and `peer_advertise` differs per
# pod.
peer_bind = "0.0.0.0:9000"

[worker]
enabled = true
# The training kinds; "all" would also claim every other compiled kind.
kinds = ["fine_tune", "graph_fine_tune", "context_predictor"]
# One rank per [gpu] devices entry: a claimed job with world_size <= 2 runs
# both ranks in this pod over a Local gang (no peer dial); a wider job makes
# this pod rank 0 of a Peer gang whose other rank is the StatefulSet's other
# replica.
local_ranks = 2

# The Ballista scheduler role, ONE dedicated single-replica Deployment: it
# hosts `[ballista] scheduler_bind` (claims and PLACES a training job onto
# a registered compute-pod executor) and `[worker] enabled = true` for the
# training kinds
# (so it can also claim and run one in-process, K4, when placement is not
# available -- e.g. no compute executor is registered yet). CPU image: this
# pod never itself trains on a device, it only decodes/re-encodes plans and
# assembles gangs.
#
# A plain (non-headless) Service in front of it is enough: a single replica
# behind a stable Service DNS name (`jammi-server-scheduler`) survives a pod
# restart exactly as well as a StatefulSet's ordinal identity would, since
# there is only ever one ordinal.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jammi-server-scheduler
spec:
  replicas: 1
  selector:
    matchLabels: { app: jammi-server-scheduler }
  template:
    metadata:
      labels: { app: jammi-server-scheduler }
    spec:
      # Same DRAIN/RELEASE shutdown contract as every other role (README
      # "Shutdown: DRAIN and RELEASE"); this pod holds no GPU, so its own
      # grace only needs to cover an in-flight in-process claim, never a
      # placed gang's device work (that grace is the compute overlay's).
      terminationGracePeriodSeconds: 600
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532
      containers:
        - name: jammi-server
          # CPU image: the scheduler role decodes/encodes plans and places
          # tasks, it never itself runs a device kernel.
          image: ghcr.io/f-inverse/jammi-ai-server:latest
          ports:
            - { containerPort: 8081, name: flight }
            - { containerPort: 8080, name: health }
            - { containerPort: 9000, name: peer }
            - { containerPort: 50050, name: ballista-sched }
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          envFrom:
            # Same catalog/broker Secret as every other role -- one shared
            # cluster, one catalog.
            - secretRef: { name: jammi-server-secrets, optional: false }
          volumeMounts:
            - { name: config, mountPath: /etc/jammi, readOnly: true }
            - { name: scratch, mountPath: /var/lib/jammi }
      volumes:
        - name: config
          configMap: { name: jammi-scheduler-config }
        - name: scratch
          emptyDir: {}

# No `type:` -> ClusterIP: reachable from the compute StatefulSet's pods
# (they dial `jammi-server-scheduler:50050` as their `[ballista.executor]
# scheduler_address`, and this Service's stable DNS name doubles as this
# pod's `[server] peer_advertise`, jammi-scheduler.toml), never externally.
apiVersion: v1
kind: Service
metadata:
  name: jammi-server-scheduler
spec:
  selector: { app: jammi-server-scheduler }
  ports:
    - { name: flight, port: 8081, targetPort: 8081 }
    - { name: health, port: 8080, targetPort: 8080 }
    - { name: peer, port: 9000, targetPort: 9000 }
    - { name: ballista-sched, port: 50050, targetPort: 50050 }

# env wins over this file (the JAMMI_<PATH> layer resolves after it); secrets
# never here -- see base/jammi.toml's header.

[storage]
result_root = "s3://jammi-results/prod"

[storage.cloud.s3]
region = "us-east-1"

[lease]
duration_secs = 30
heartbeat_secs = 10

[ballista]
# This pod hosts the ONE Ballista scheduler: it claims a training job and
# PLACES it on a registered compute-pod executor (configuration.md's
# semantics: this table's presence makes a process a scheduler). No
# `[ballista.executor]` here -- this pod never itself registers as an
# executor, so a task it places can never be bound back to it (a task's own
# submitter is excluded from placement in any case).
scheduler_bind = "0.0.0.0:50050"

# This pod claims the training kinds a placed run handles (`fine_tune`,
# `graph_fine_tune`) and places each claimed job on a compute executor; while
# no compute executor is registered it runs the job in-process, exactly like
# any other worker-enabled process (the bytes are identical either way).
# `context_predictor` has no placed arm, so it is left to the compute pods:
# listing it here would train it on this CPU pod.
[worker]
enabled = true
kinds = ["fine_tune", "graph_fine_tune"]

# This pod's own peer/gang listener -- needed because a training job this
# pod claims and runs in-process (no placement available) may still assemble
# a multi-host gang across the compute StatefulSet's pods.
[server]
peer_bind = "0.0.0.0:9000"
# The stable Service DNS name this pod is always reachable at (a single
# replica behind a normal ClusterIP Service never needs a downward-API pod
# name the way the compute StatefulSet's per-pod `peer_advertise` does).
peer_advertise = "jammi-server-scheduler:9000"
# Pure scheduler node: mounts no query-tier gRPC service.
services = []

Both :latest tags are re-pointed by every v* release tag (never by a prerelease); the CPU :latest can additionally be re-pointed to the current main by a manual build-and-push-main dispatch. Pin an exact :vX.Y.Z tag for reproducible GPU-node deploys.

Very high scale, specialized GPU pools, and a split compliance posture (query tier vs. training tier on separate node pools / network policies) are the shapes this topology serves.

Beyond-one-node retrieval

A query-tier replica answers Search over a table whose ANN index segments it does not all hold by fanning the query out to the replicas that own them: each owner searches its segments and returns (row_id, distance) hits — never vectors — and the coordinator merges them under the same total order a single node merges its own segments with (the distributed data plane; see Security Posture for the I-PEER invariant and Operability for the failure ladder). Three facts fix the shape:

  • The default is AllLocal. With [server] peer_bind unset — every shape above — every segment is this replica’s, there is no third listener, and the search is exactly the single-node search (the same kernels, the same bytes, the same exact-read count at every segment count). Nothing on this page changes until a deployment opts in.
  • peer_bind makes a replica an owner (and a gang admission member). Setting [server] peer_bind (JAMMI_SERVER__PEER_BIND) opens the internal listener serving both PeerService (segment search) and GangService (multi-host training-run admission — see Security Posture) on that replica. Bind it on a private interface behind network policy / mTLS from the runtime — its clients are other jammi coordinators and it authenticates nothing itself.
  • Precondition: a shared, replica-readable result_root. A segment an owner serves must be a bundle every replica can reach: [storage] result_root on an object store every replica reads (a local artifact_dir is one node’s). Placement — which replica owns which segment — is derived at query time from the live replica ring ([server] peer_advertise + the catalog’s instances rows, rendezvous- hashed over (table, segment)), never declared; the membership half of that ring is the compute-tier substrate’s (peer_advertise, instances.peer_addr), and until it lands a library process supplies an explicit StaticPlacement through InferenceSession::open_with_placement. Batch builders (the neighbor graph, eval) never fan out: they load the whole table’s segment set on the building replica.
  • Membership is judged fresh under a margin, pruned only well beyond it. A row missing its heartbeat for 2 × lease (instance_liveness_margin) is no longer a fresh member; it is not actually deleted until 3 × lease (instance_prune_window) — strictly beyond the margin, so a merely-stale member still has its row when the lease keeper’s own reregister lands. A process whose row WAS pruned during a transient outage (an outage longer than the window) rejoins the ring on its very next successful heartbeat, with no restart: the keeper’s reregister re-upserts the whole membership tuple (the instances row and, if this process runs a claim loop, its workers row) in one transaction. instances.result_root records the configured root SPELLING verbatim, but neither list_gang_members (the gang-membership listing verb, contract feat_500-C-U5b-1a §12) nor this ring consults it in this unit — root identity across spellings, and any membership predicate built on it, is a separate, not-yet-built unit. Were root-string equality ever made part of a membership predicate here, it would still be necessary, never sufficient, for shared storage — the attestation VERIFY a later unit owns is what establishes sufficiency.

A REFRESHED table’s Mixed arm is not version-aware. The single-node (AllLocal, every segment this replica’s own) search path always resolves a versioned table’s CURRENT version before searching it. The multi-node Mixed arm — reached only when peer_bind is set and this table’s segments span more than one replica — does not: it plans off list_index_segments’ flat, unversioned segment set, the same limitation the single-node path carried before its own version-aware resolution was added. If a Shape D deployment places a table refresh_embeddings has since published a new version of across more than one owning replica, a Search served through peers can surface rows from a version older than the table’s current one; keep a refreshed table’s segments on a single owning replica (or force-local it) until this closes.

The jammi-server probe subcommand

Every shape above that runs jammi-server — B, C, D — uses the same readiness mechanism: jammi-server probe [--url URL] [--config PATH] [--timeout-secs N] GETs /readyz once and exits 0 on HTTP 200, 1 otherwise (never following a redirect), printing the failure status and body to stderr. Without --url, the target derives from the resolved [server] health_listen through the identical config-resolution chain serve uses. It is a generic CLI a supervisor drives from a subprocess exit code — Compose’s HEALTHCHECK, systemd’s ExecStartPost=, Nomad’s check { type = "script" } — the same shape as Postgres’s pg_isready. Kubernetes needs none of this: its readinessProbe.httpGet already speaks HTTP directly against /readyz.

What the published images can and cannot do

The published CPU image (ghcr.io/f-inverse/jammi-ai-server) is built with cargo build --features jammi-server/jetstream-broker,jammi-server/storage-cloud — nothing else. GetServerInfo.features on that image therefore reports ["jetstream-broker"], and storage_backends reports the schemes storage-cloud pulls in (s3, r2, gs, azure, alongside the always-on file/memory).

A Postgres catalog ([catalog.postgres], Shapes B/C/D above) works on this image unconditionally: sqlx’s Postgres driver is not gated behind any cargo feature, so it is not a features entry at all — it is simply always present.

The separate postgres cargo feature (datafusion-table-providers/postgres) is a different capability — querying a remote Postgres database as a federated SQL source, not the catalog backend — and is not compiled into the published image. It would appear in features as "postgres" only on a custom build that opts into it explicitly; do not read this image’s Postgres-catalog support as evidence that source federation is available.

Node architecture. The CPU image’s generic tags (ghcr.io/f-inverse/jammi-ai-server:latest/:vX.Y.Z/:vX.Y and their sha-<sha> equivalents) are a multi-arch index — linux/amd64 and linux/arm64 — so Shapes B/C’s query tier and Shape D’s disaggregated query tier can schedule onto either an amd64 or an arm64 node pool without a per-arch tag; docker pull/Kubernetes resolve the right member automatically. Shape D’s GPU compute tier is unaffected by this: the CUDA image (-cu12) is linux/amd64 only, so the GPU node pool stays amd64. The same CPU image name’s self-contained tags (:selfcontained, :selfcontained-sha-<sha>) are also linux/amd64 only — never schedule those onto an arm64 node pool.