Security Posture

Jammi is an engine on a trusted network, not a security boundary. This page is the published threat model: precisely what the engine defends, what it explicitly does not, the trusted-network assumption every deployment inherits, and the consumer’s responsibilities for the boundary the engine deliberately does not own. Every “defends” line below traces to a real test or code path; every “does not” line is the honest absence of a guarantee the engine never claims.

The single principle, stated once and not softened: Jammi authenticates nothing. Identity, authorization, and the network perimeter are a consumer’s vocabulary; the engine ships the seam a consumer plugs them into, never the policy. A deployment that exposes the engine’s port to an untrusted caller has removed the boundary the engine assumes is there.

What the engine defends

Each line names the mechanism and the test or code path that proves it.

The BYO-auth seam’s contract is pinned by grpc_byo_auth.rs as a worked example. The resolver-seam tests above prove it through the engine’s own composability seam across both transports; the custom-interceptor-in-front form (a consumer fronting its own single service) additionally pins four guarantees, each its own test:

• Missing credential → unauthenticated. No token fails the request before any handler runs, so the caller reads nothing — it does not fall through to an unscoped read (missing_credential_is_rejected_not_run_unscoped).
• Forged claim → unauthenticated. A token whose signature does not cover its tenant claim is rejected; a forged tenant buys nothing because the signature covers the claim (invalid_credential_is_rejected).
• A rejected caller does not fall through. The interceptor fails the request rather than binding None — there is no path by which an unauthenticated caller silently reads another tenant’s rows (the same test proves a valid token for the very tenant the forgery claimed does resolve, so the rejection was the signature, not a tenant blocklist).
• Per-tenant isolation through the seam. Two callers presenting valid tokens for two distinct tenants each see only their own tenant’s sources, end to end through the authenticating interceptor (two_authenticated_tenants_see_isolated_sources).

What the engine explicitly does NOT defend

These are honest absences. The engine never claims them; a deployment that needs them supplies them above the engine.

• It authenticates nothing. There is no built-in credential check on any verb. The default SessionIdTenantResolver reads the jammi-session-id header and binds the tenant the caller asserts (or the explicit Global/unscoped scope when none is bound) — it verifies nothing about who the caller is. A deployment that needs authentication supplies its own TenantResolver at the seam.
• It ships no authz / RBAC / SSO. There is no role model, no permission check, no policy engine, no identity-provider integration. Authorization is a consumer’s vocabulary and lives above the seam.
• jammi-session-id is a correlation id, NOT a credential or principal. It is a client-minted, opaque transport correlation id identifying a connection, not a person. Anyone who presents another session’s id assumes that session’s tenant. It is never an authentication or authorization boundary.
• No TLS / secrets / IAM. Transport encryption, secret management, key rotation, and cloud IAM are the consumer’s runtime, not the engine’s — the same line the Design Philosophy draws around load balancing, ingress, and orchestration.
• The peer listener authenticates nothing either (I-PEER). [server] peer_bind (unset by default) serves the engine-internal segment-search seam to other replicas and trusts the channel — see The peer listener below.
• The gang listener authenticates nothing either (I-GANG). The same [server] peer_bind listener also serves the coordinator-to-member gang admission seam for a multi-host training run and trusts the channel — see The gang listener below.
• The Ballista listeners authenticate nothing either (I-PEER). [ballista] scheduler_bind and [ballista.executor] bind/grpc_bind (all unset by default) open the compute plane’s scheduler gRPC, executor task gRPC and Arrow Flight shuffle ports and trust the channel — see The Ballista listeners below.

The peer listener (I-PEER)

[server] peer_bind opens a separate internal listener serving jammi.v1.peer.PeerService — the seam a coordinator replica fans a search out through to the replicas that own a table’s segments (see Beyond one node). Its threat model is stated as one invariant, I-PEER:

• Every client of peer_bind is a jammi coordinator. The owner handler trusts the channel: the request carries no tenant, the owner binds none and reads no result_tables row. It enforces exactly one thing at its input edge — every requested segment id belongs to the named table (else the whole request is refused) and the bundle’s stamped precision matches.
• Tenant scope is enforced once, at the coordinator. The coordinator’s Search resolved the table through its own tenant-scoped catalog read (tenant_id = $current OR tenant_id IS NULL) before any fan-out, so a coordinator bound to tenant B cannot name tenant A’s table — it fails before a single peer call. The owner is the second half of that one predicate, not a second predicate.
• The public listener never reaches it. The peer routes are built outside assemble_grpc_chain, never wrapped by the tenant-binding layer, never advertised by GetServerInfo; the public listener answers UNIMPLEMENTED for /jammi.v1.peer.PeerService/* (proven by the tenant-isolation oracle).
• Binding peer_bind on a routable interface without network policy / mTLS exposes cross-tenant segment reads to anyone who can reach the port. The listener speaks plaintext gRPC like every other engine port; encryption and peer authentication are the runtime’s (a mesh, a network policy, mTLS at a sidecar), exactly as for the public listener. Default unset = no listener.

The gang listener (I-GANG)

The same [server] peer_bind listener also serves jammi.v1.gang.GangService (RunRank) — the seam a coordinator uses to admit a member into a multi-host training run. Its threat model is stated as one invariant, I-GANG:

• Every client of peer_bind is a jammi coordinator, the same trust I-PEER states — no separate authentication for the gang seam.
• Tenant is derived from the job row, never accepted from the caller. A RunRank call carries only job coordinates (job_id, attempt, rank, world, coordinator_instance_id); caller-supplied tenant metadata is never read, and ambient admin scope is refused before any row is read (and again at the training-set resolution site). The admission row the member reads (Catalog::get_job_for_rank, primary key only) carries the row’s OWN tenant_id, and a multi-host job’s training set is resolved under that tenant alone through the strict tenant-pinned resolver (Catalog::get_result_table_for_tenant) — another tenant’s table of the same name, or a global (NULL-tenant) one, never resolves. The job’s coordinates are the capability: a coordinator on peer_bind holding them is admitted for that job — bounded by I-PEER’s own trust statement above — and can read nothing beyond what that job’s row names.
• Non-disclosure on refusal; reasons only after admission. Every admission determinant — job not found, not running, wrong claimant, wrong attempt, lease not live or undecodable, world size undecodable or not the caller’s, the training-set pair missing, the tenant undecodable, the training set unresolved under the job’s tenant / not ready / its sidecar absent / its digest mismatching / this host’s store faulting, coordinator not fresh — collapses to the SAME status (FAILED_PRECONDITION) with a fixed message, as the call’s own result: the listener discloses neither a job’s existence, its claimant, its attempt, its tenant, nor another tenant’s table; tests distinguish determinants through a test-only seam, never response text. Only an ADMITTED session — the caller already holds the job’s own coordinates — ends with a named reason in the stream (Aborted{Drain | Refuted | Cancelled | NoBody | StoreUnavailable | Unavailable}).
• Admit-and-hold. An admitted rank holds this host’s single job slot (a peer never claims while it holds a rank, never admits a rank while it runs a job; a busy slot refuses UNAVAILABLE, transient) and is re-verified every heartbeat against the same determinants; a DRAIN or RELEASE of the host ends every held rank at once. A world_size > 1 session runs its rank body (trained over the session’s own stream as a rank of the gang) and ends with that body’s one Outcome; a world_size == 1 session has no body to run, parks, and ends NoBody after one lease window. The member writes nothing to the job row and nothing into the job’s artifact prefix on behalf of a rank: only the lease holder writes.
• Multi-host gang admission is a Postgres-only deployment shape — a SQLite deployment is a single embedded process; there is no second host to admit at all, so this seam is never exercised there. The admission read’s lease/freshness facts (lease_expires_at, instances.last_seen_at) are decoded CLIENT-SIDE from the row’s raw stored text (jammi_db::catalog::lease::{decode_lease_expires_at, last_seen_at_is_fresh}), never a SQL-side cast, so a malformed value is a row fact the handler refuses on EITHER backend, never a read fault on one and a live-row fact on the other (https://github.com/f-inverse/jammi-ai/issues/574); the CLAIM and RECLAIM predicates (Catalog::claim_next, Catalog::reclaim_expired_jobs) are the ones that still require a shared, single-writer clock (Postgres’s now()) — those WRITE, and a wrongly-reaped live claimant is destructive in a way a stale admission read (self-correcting at the next heartbeat) is not.
• The public listener never reaches it. The gang routes are mounted beside PeerService outside assemble_grpc_chain, never wrapped by the tenant-binding layer, never advertised by GetServerInfo; the public listener answers UNIMPLEMENTED for /jammi.v1.gang.GangService/* (proven by the tenant-isolation oracle).
• Binding peer_bind on a routable interface without network policy / mTLS exposes the same risk I-PEER states — the listener speaks plaintext gRPC; encryption and peer authentication are the runtime’s. Default unset = no listener = no gang admission surface.
• The membership predicate list_gang_members narrows to is a catalog-local read, not an RPC. A row is a member of a listing iff: it is NOT the caller itself; its workers row (an INNER join — a member is a fleet worker with a claim-loop slot, not merely a live process) has state = 'claiming' (warming/draining excluded); workers.kinds contains the requested kind as a whole, comma-split, trimmed token (never a substring — fine_tune never matches graph_fine_tune); instances.peer_addr is non-NULL; and last_seen_at is fresh under instance_liveness_margin(lease) (2 × lease) on the DB clock; and its instances.result_root_identity EQUALS the caller’s own (unit U5b-1a-A2). peer_addr_of is the by-id analogue, with no kind/self/root filter (any other member may resolve any other by id). The root is compared by identity, never by spelling: instances.result_root carries resolved_result_root() VERBATIM (so gcs://b/p and gs://b/p remain two different STRINGS in that column), and result_root_identity carries that root’s identity, derived by MemberRoot::resolved from the same config the store roots itself from: the store’s own URL parser folds the scheme aliases (gcs://→gs://, abfss://→azure://), the bucket is kept as spelled (the driver dials it as spelled) and the key goes through the store’s own key parser (one leading / stripped, a trailing one dropped, an empty segment refused — keys keep their case), the location determinants are read back from the very builder the store constructs for that root (storage::location_determinants: the process environment first, [storage.cloud] on top — for S3/R2 the bucket endpoint the driver dials (endpoint spelling, virtual-hosted or path style, S3 Express, region), the Azure account, endpoint, Azurite host in emulator mode, Fabric switch, the GCS base URL; the identity spells exactly the variables the driver spells and reads their values as the driver reads them, so nothing the driver honours can be missed), a local root is created (as the store creates it at open) and canonicalised on the owning host’s filesystem (symlinks, ./.., the filesystem’s own spelling), and an in-memory root is refused at registration as unshareable. Two members whose spellings name one location ARE gang members of each other; two rooted at different locations are not; a row with a NULL identity (pre-036, or a process with no membership) never matches. Root identity equality is NECESSARY, never SUFFICIENT, for shared storage: two identical local roots on two unshared filesystems are indistinguishable to this predicate — sufficiency is the attestation VERIFY’s (U5a-1’s admission-time sidecar, U5b-0’s leaf inventory), not this listing’s.
• B5 (this listing is deliberately tenant-free). instances/workers rows are deployment infrastructure (which processes exist, what they claim, where they are reachable), never tenant data — there is no tenant column on either table, so there is no tenant predicate to drop or keep, and list_gang_members/peer_addr_of return the identical answer under any tenant scope and under none. Neither verb is reachable from any RPC at all today (no gang/peer handler calls either) — a vacuous truth, not yet a tested boundary; the first RPC that calls one of them owes the enumerating unreachability-or-scoping oracle this listener’s own precedent (GANG_LISTENER_ALLOWLIST/PEER_LISTENER_ALLOWLIST) already sets.

The Ballista listeners (I-PEER)

A process with a [ballista] role opens up to three more internal listeners, none of them on the public tenant layer: the scheduler’s gRPC ([ballista] scheduler_bind, Ballista’s SchedulerGrpc), the executor’s task gRPC ([ballista.executor] grpc_bind, ExecutorGrpc) and the executor’s Arrow Flight shuffle port ([ballista.executor] bind). They serve Ballista’s own wire package, not jammi.v1, and their threat model is the peer listener’s, I-PEER:

• Every client of these ports is a jammi role. A scheduler is dialled by the executors it places on and by the scheduler-role process’s own submitter; an executor is dialled by its scheduler and by sibling executors for shuffle reads. The listeners carry no tenant of their own and bind none.
• Tenant scope is enforced once, at the submitting session. The plan a scheduler places carries the submitter’s tenant inside the operator descriptor (JammiCodec writes it; a plan the codec did not encode is refused by magic). An executor rebuilding a result-table read resolves the table through the strict tenant-pinned catalog read under exactly that tenant — never the executor process’s ambient scope, which a scheduler or executor process does not have. A descriptor naming another tenant’s table is refused with the same non-disclosing not-found the public layer answers.
• The public listener never serves them. No Ballista service is mounted on assemble_grpc_chain’s routes; the tenant-isolation oracle proves the public listener answers UNIMPLEMENTED for SchedulerGrpc, the same way it does for PeerService and GangService.
• Binding them on a routable interface without network policy / mTLS exposes the compute plane — task submission, shuffle reads, executor registration — to anyone who can reach the ports, exactly as for peer_bind. The shape-d overlay binds them on the pod network and the deployer owes the network policy that keeps them cluster-internal; encryption and peer authentication are the runtime’s (a mesh, mTLS at a sidecar). Default unset = no listener.

Transport encryption is the deployer’s runtime, not the engine’s

The engine speaks plaintext gRPC and Flight SQL and ships no TLS code path; transport encryption is the deployer’s runtime. It follows from the same primitives this page and the Design Philosophy already state:

• B4 (“one binary, every topology”) constrains what the engine forks on, not what fronts it. Terminating TLS is supplied by the runtime the engine deploys into — a proxy, a mesh sidecar, a load balancer — and that termination is not a topology-specific code path the engine would need to special-case per shape. A tls cargo feature would itself be the kind of server-only gate B4 refuses: a build-time fork between “the engine” and “the engine, but for a server.”
• Passing the discipline test is necessary, not sufficient. A user who has never heard of any consumer does want the wire encrypted — TLS passes the discipline test on its own. But the boundary table and the paragraph that follows it (docs/guide/src/philosophy.md:116-125) are the second gate: TLS, secrets, IAM, ingress, and load balancing all pass the discipline test and are still placed in the consumer’s runtime, not the engine, because the table asks a second question the discipline test does not — does owning this turn the engine into infrastructure it isn’t. TLS termination answers yes.
• A [server] tls key, in the file or the environment, is a typed refusal, not a silent no-op. ServerConfig — what [server] deserializes into — is a #[serde(default, deny_unknown_fields)] struct (crates/jammi-db/src/config/mod.rs:1123), the same discipline JammiConfig itself carries at its top level (crates/jammi-db/src/config/mod.rs:204). A [server] tls = … stanza in a config file, and JAMMI_SERVER__TLS in the environment, are not silently ignored — each is a typed JammiError::Config startup refusal, naming the unrecognised key.
• There is no engine-side certificate to hand the TenantResolver seam. Because termination happens outside the engine, the engine never sees a peer certificate to map onto a tenant — that mapping, if a deployment wants one, lives in the terminator or the proxy in front, not at the seam described under The identity seam. The CLI’s --target refuses grpcs:// and https:// with a typed error naming the accepted schemes (crates/jammi-cli/src/main.rs:170-187; the CHANGELOG’s “drop grpcs:// and https:// as accepted --target schemes” entry (#480), commit 616bb6d4) rather than advertising a transport it cannot speak — put a TLS-terminating proxy in front and point --target at it in plaintext (grpc:///http://). This is an asymmetry between the two clients: the Python SDK’s RemoteTarget legitimately keeps grpcs:///https:// in its own scheme table (clients/python/jammi/_target.py:51-54) because it is a general client library reaching whatever endpoint a deployment publishes (including a TLS-terminating proxy), while the CLI is the engine’s own admin surface and names only the schemes the engine itself speaks.

Shape B with no mesh — an on-prem single-tenant deployment that has no ingress or mesh to terminate TLS for it — still gets encryption: put a terminator in front of the engine’s plaintext listeners. A minimal, consumer-neutral example with Caddy:

# Caddyfile — terminates TLS and forwards plaintext to the engine's
# loopback-bound listeners (see docker-compose.yml).
#
# `tls internal` issues Caddy's own locally-trusted certificate: a private
# DNS name (no public record) has no ACME challenge path to a public CA,
# so automatic Let's Encrypt/ZeroSSL issuance is not an option here.
jammi.example.com {
    tls internal
    reverse_proxy h2c://127.0.0.1:8081  # gRPC + Flight SQL
}
health.jammi.example.com {
    tls internal
    reverse_proxy 127.0.0.1:8080        # /healthz, /readyz, /metrics
}

The reference deploy/docker-compose.yml binds its published ports to 127.0.0.1 for exactly this shape: the compose stack publishes the engine’s ports for a terminator running on the same host to reach, not for direct exposure to an untrusted network. A terminator running as a container on the same Compose network instead reaches the engine by its service name rather than 127.0.0.1 — reverse_proxy h2c://jammi-server:8081 — since two containers on the same Compose network share that network, not the host’s loopback interface.

The trusted-network assumption

Every Jammi deployment that uses the default SessionIdTenantResolver assumes a trusted network: a private VPC, a sidecar mesh, or a single-process embedding where every caller is already inside the trust boundary. On that network, binding the tenant a caller asserts via jammi-session-id is the right, low-friction trade-off. The moment an untrusted caller can reach the port, that trade-off is wrong — and closing it is the consumer’s job, via an authenticating TenantResolver at the seam, not a flag the engine flips.

Tenant scope is an organizational mechanism, not an access-control boundary

This distinction is load-bearing. Tenant-scope filtering (above) is an organizational mechanism: it keeps one tenant’s catalog rows from appearing in another tenant’s correctly-bound reads, so a multi-tenant deployment stays tidy and a buggy caller that writes the wrong tenant_id is refused by assert_tenant_matches. It is not an access-control boundary: it does not decide which tenant a caller is entitled to act as. Nothing in the engine prevents an unauthenticated caller from asserting any tenant it likes via jammi-session-id and reading that tenant’s rows. Access control — proving a caller may act as the tenant it claims — is exactly what the BYO-auth seam adds in front of the scope mechanism. Treating tenant scope as if it were authorization is the misuse this page exists to forestall.

The consumer’s responsibilities

To put a real tenant boundary in front of untrusted callers, a consumer supplies the authentication and authorization the engine deliberately omits by implementing a TenantResolver and passing it to assemble_grpc_chain — one plug that binds every engine gRPC verb and the Flight db.sql lane through the single tenant-binding mechanism (the tenant recipe and the worked example in grpc_byo_auth.rs):

1. Authenticate the principal. In resolve, read and verify the caller’s credential (a bearer token, an exchanged session cookie, a service-to-service token). A missing or invalid credential returns Err(Status::unauthenticated) here, before any handler runs.
2. Authorize the tenant from the verified claim. Derive the tenant from the verified claim — never from a header the caller controls. This is where the consumer’s policy lives: which tenant this principal may act as. Return Ok(TenantScope::Tenant(t)).
3. The engine binds it. The async tenant-binding layer maps the resolved scope onto the SessionTenant request extension every verb handler resolves, and TenantBoundProvider binds it for Flight — the consumer writes only resolve.

Because resolve runs in front of every handler, the tenant the engine acts on is the one the credential proves, not one the caller asserts. Reject, don’t default: an authenticating resolver returns Tenant/Err and NEVER TenantScope::Global — returning Global (or, in the lower-level interceptor-in-front form, binding None) on a failed check runs the request unscoped, which for a tenant_id IS NULL-bearing catalog is a global read, so a rejected caller must fail the request. TenantScope::Global is the explicit unscoped choice the default (OSS-cooperative) resolver returns when no tenant is bound — never a value a rejection falls through to. That framing rule is the defect grpc_byo_auth.rs’s missing_credential_is_rejected_not_run_unscoped and the seam mirror resolver_seam_binds_the_engine_and_rejects_missing_credential guard against.

Dependency-advisory posture

The engine’s dependency tree is gated in CI by cargo deny against the RustSec advisory database, plus a license allowlist and the source/ban guards that formalize the engine’s one-way dependency direction (no proprietary or non-crates.io crate in the OSS closure). The advisory lane runs the live RustSec DB on every PR; a documented exception in deny.toml records any advisory the release knowingly carries, with a written rationale, rather than destabilizing the freeze with a risky bump. The config is deny.toml at the repo root.