Format Stability

Jammi persists several on-disk formats. Each one is stamped with a version the writer records and the reader checks, so a file written by a newer build — or by a backend whose serialized layout changed — is rejected as a typed error rather than silently misparsed into wrong data. This page is the operator’s reference for every persisted format the engine owns, what its stability stamp is, and how a reader reacts to a stamp it cannot honour.

The single principle: a reader never guesses. When a stamp is unreadable the load fails loud with a typed error; the upgrade path is to re-emit the artifact from its definition. There is no back-compat reader, no silent downgrade, no default-to-version-1. (Re-emitting is cheap and exact on the producing host: a result table is the deterministic output of its producing definition over its pinned input anchors — see The Materialization Contract. Across hosts a CPU-produced result table’s identity is the catalog row plus its definition_hash, not its bytes: a float reduction can move by an ULP between hosts of different CPU microarchitecture even though the definition and inputs are pinned identically.)

The per-format table

Two distinct kinds of stamp appear above, and the difference is deliberate:

• Reject-newer for formats that carry a compatibility ordering: only a newer version carries a layout this build does not know, so only a newer version is refused on the stamp. The .rowmap and ANN .manifest.json are this kind (found > ROWMAP_VERSION, found > ANN_MANIFEST_VERSION).
• Exact-version for the materialization manifest (MaterializationManifest::from_json_bytes), whose version names a determinant set rather than a layout: an older version is a superseded determinant set, so a body that still decodes is refused on any inequality (found != MANIFEST_VERSION), never trusted as a hit. The decode runs before the stamp is inspected, so a body that lacks a field the current shape requires — a producer variant that grew a required determinant at the same version, as the fine-tune descriptor’s topology fields did — is the typed ManifestError::Serde; the one older shape the reader names on its own (an object at the current version with no leaves) is read as absent.
• Strict for the USearch backend_version, because the USearch serialized graph format carries no compatibility ordering between releases. A version that differs at all may mis-deserialise the graph and return wrong neighbours, so any inequality is incompatible — there is no “older is fine” here.

Materialization manifest — exact-version

.materialization.json carries manifest_version (MANIFEST_VERSION). The reader rejects any other version, older or newer, as the typed ManifestError::UnsupportedManifestVersion { found, supported } — the version names the determinant set the hash was computed over, and a superseded set is never a hit. The version is checked after the decode and before the parsed manifest is handed back. Every other stamped format is modeled on this fail-loud shape (the ordered ones relax it to reject-newer); the full contract is in The Materialization Contract. Its error lives in its own domain (ManifestError) and is intentionally not folded into the shared IncompatibleFormat variant — it carries the manifest-specific recovery semantics the contract describes.

ANN row map (.rowmap) — reject-newer

The .rowmap is the engine-owned mapping from a USearch internal id to the Jammi _row_id string. It is a small binary file: a leading u32 version header, then length-prefixed UTF-8 entries. On load the reader checks the header and rejects a version greater than ROWMAP_VERSION as JammiError::IncompatibleFormat { artifact: "rowmap", found, supported }.

ANN sidecar manifest (.manifest.json) — reject-newer + strict backend

The ANN sidecar’s .manifest.json records the index metadata: version, dimensions, backend, backend_version, scalar_kind, count, the file names, and the creation instant. On load it is deserialised as a typed struct (mirroring MaterializationManifest::from_json_bytes), never by field-by-key serde_json::Value lookups. The determinants of a safe load — version, dimensions, backend_version, and scalar_kind — are all required: a manifest missing any of them is a hard decode error, never silently defaulted to a guess. A fifth field, binary_threshold_kind, is conditionally required: legitimately absent for every non-Binary scalar_kind, but its absence on a Binary manifest with at least one row is itself a hard error — a torn or pre-threshold-fix bundle, not a legitimate empty state.

Two checks run on the deserialised manifest:

1. version, reject-newer. A version greater than ANN_MANIFEST_VERSION is JammiError::IncompatibleFormat { artifact: "ann-manifest", .. }.
2. backend_version, strict. The stamped USearch version is compared for exact equality against the linked jammi_db::index::backend_version(). Any mismatch is JammiError::IncompatibleFormat { artifact: "usearch-index", .. }.

A Binary scalar_kind additionally loads the .threshold companion beside the bundle — the per-dimension threshold τ the sidecar’s sign-packing was fit against — validated against dimensions and the manifest’s binary_threshold_kind rather than a stamp of its own (see the per-format table above).

USearch ANN graph (.usearch) — strict backend version

The .usearch file is USearch’s own serialized HNSW graph. Its serialized header carries only the major version and gives no cross-release compatibility guarantee, so a USearch upgrade can change the on-disk layout in a way that deserialises into a structurally-valid-but-wrong graph — returning incorrect nearest neighbours with no error. To close that silent-corruption path, the engine stamps the full linked USearch version (backend_version) into the sidecar .manifest.json at save and strict-compares it on load. The graph itself is never trusted across a backend version change; the only safe action is to re-emit the embedding table (which rebuilds the sidecar).

Lexical (BM25) index — library-loud

The lexical retrieval sidecar is a tantivy index directory. Tantivy stamps its own format version and refuses to open an index written by an incompatible release: Index::open returns IncompatibleIndex, which the engine surfaces as JammiError::Lexical. The engine adds no stamp of its own here — the library is already loud, so a second stamp would be redundant machinery. The recovery is the same: re-emit (re-index) the table.

Result-table Parquet — no embedded stamp, by design

The result-table Parquet object carries no embedded format version. Its format-of-record version is the manifest_version of the .materialization.json sidecar written beside it: the manifest is the artifact’s identity, and the Parquet bytes are its subject. A reader does not need a second, in-band version because:

• Shape safety is enforced at read time by the typed Arrow downcast. Every vector read goes through store::vectors::extend_with_fixed_size_list_f32, the single place in the engine that downcasts a vector column to FixedSizeList<Float32>; a missing column, a wrong Arrow type, or a non-Float32 inner type is a typed JammiError::Schema, not a panic. Schema shape is checked from the data itself, so it needs no stamp.
• Byte integrity is content-addressed. The Parquet object is immutable and identified by its ArtifactDigest (SHA-256 over the bytes) recorded in the manifest, so any out-of-band byte change is caught by verify_materialization recomputing the digest — see The Materialization Contract.

The manifest-bypass Parquet read paths

Three engine read paths open a result-table Parquet object directly, without first reading the .materialization.json manifest:

• Session::read_vectors — streams the whole vector column of an embedding table into one Vec<f32> per row.
• Session::read_vector_by_key — extracts a single row’s vector by its _row_id (the resolver behind search_by_id’s query-by-example path).
• store::register_parquet_table — registers a Parquet URL as a DataFusion table under jammi.{name} for SQL scans.

These paths do not consult a format stamp, and that is correct: their safety rests entirely on the typed JammiError::Schema downcast in store::vectors::extend_with_fixed_size_list_f32, which validates the on-disk Arrow schema shape directly from the data. A Parquet object whose schema does not match what the read expects produces a typed Schema error, regardless of how it was written. Out-of-band byte tampering on these immutable, content-addressed objects is the verify_materialization digest check’s concern, not a per-read stamp’s.

Upgrade path: re-emit

For every stamped format above, the recovery from an incompatible stamp is the same — re-emit the artifact from its definition. The engine ships no back-compat reader and no in-place migrator: an ANN sidecar is rebuilt by re-running the embedding producer, a tantivy index by re-indexing, a result table by re-running its producing definition over its input anchors. A result table’s identity after re-emission is its catalog row and definition_hash, which are exact and not lossy; on the producing host the bytes are exact too, but a re-emission on a different CPU host is not asserted byte-identical to the original (see The Materialization Contract). The typed rejection is the signal to re-emit; it is never something to paper over with a default.