Generate Embeddings

Measured companion: for the long-form, executed-and-measured Python treatment, see The Cookbook → Constructing the Graph.

Generate vector embeddings by running a model over text columns from a registered source. Results are persisted to Parquet with sidecar ANN indexes for fast similarity search.

Basic usage

Rust

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use jammi_ai::session::InferenceSession;
use jammi_db::store::CachePolicy;
async fn ex(session: &InferenceSession) -> jammi_db::error::Result<()> {
let (record, _outcome) = session.generate_text_embeddings(
    "patents",
    "sentence-transformers/all-MiniLM-L6-v2",
    &["abstract".to_string()],
    "id",
    CachePolicy::Bypass,
    None,
).await?;

println!("Embedded {} rows, {} dimensions", record.row_count, record.dimensions().unwrap());
Ok(()) }
}

Python

db.generate_embeddings(
    source="patents",
    model="sentence-transformers/all-MiniLM-L6-v2",
    columns=["abstract"],
    key="id",
    modality="text",
)

What gets created

Each call creates a timestamped Parquet file plus a sidecar ANN index bundle:

{artifact_dir}/jammi_db/
├── patents__embedding__all-MiniLM-L6-v2__20260325T120000.parquet
├── patents__embedding__all-MiniLM-L6-v2__20260325T120000.usearch
├── patents__embedding__all-MiniLM-L6-v2__20260325T120000.rowmap
└── patents__embedding__all-MiniLM-L6-v2__20260325T120000.manifest.json

• Parquet file — source of truth. Contains _row_id, _source_id, _model_id, vector, _content_hash. Readable by external tools (DuckDB, Polars, pandas).
• .usearch — USearch HNSW graph for ANN search.
• .rowmap — maps internal USearch keys to _row_id strings.
• .manifest.json — metadata (dimensions, count, metric, backend).

The sidecar files are disposable — deleting them falls back to brute-force exact search. The Parquet file is the only thing that matters.

Embedding table schema

Failed rows (null or empty text) are excluded — only successfully embedded rows appear in the output. Rows are written in key order: CAST(key AS Utf8) ascending, ties broken by _content_hash, so the table’s bytes are identical across engine.execution_threads.

A NULL in the key column is not a per-row failure: the whole call is refused with the typed InvalidKey { column, null_count } before the model runs (the null count is exact; zero rows are embedded and nothing is written). Every row needs a key.

_content_hash is what makes the table refreshable: after the source changes, refresh_embeddings re-embeds only the rows whose hash differs and publishes a new version of the same table — see Refresh an Embedding Table Incrementally.

Text column format

A text column can be:

• Utf8 / LargeUtf8 / Utf8View — read directly, no conversion. Utf8View is what a plain Parquet Utf8 column becomes through a DataFusion scan under this workspace’s pinned Arrow/DataFusion versions, so an ordinary registered source’s text column takes this arm on the real, unmodified scan output.
• Any other non-binary type (e.g. Int64, Float64) — cast to text ("42", "3.14", …). The whole call is refused if the cast would introduce a null the source column did not have.
• Binary / LargeBinary / BinaryView / FixedSizeBinary — refused outright, naming the column’s data type. These hold raw bytes, not text (e.g. image or audio bytes submitted under a text-embedding task by mistake); the call never silently embeds an empty string for every row.

A null value in an otherwise-text column reads as the empty string (""), which the row-failure rule above then excludes from the output.

Multiple text columns

Pass multiple column names to concatenate them (space-separated) before embedding:

Rust

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use jammi_ai::session::InferenceSession;
use jammi_db::store::CachePolicy;
async fn ex(session: &InferenceSession) -> jammi_db::error::Result<()> {
session.generate_text_embeddings(
    "papers",
    "sentence-transformers/all-MiniLM-L6-v2",
    &["title".to_string(), "abstract".to_string()],
    "doi",
    CachePolicy::Bypass,
    None,
).await?;
Ok(()) }
}

Python

db.generate_embeddings(
    source="papers",
    model="sentence-transformers/all-MiniLM-L6-v2",
    columns=["title", "abstract"],
    key="doi",
    modality="text",
)

Multiple embedding tables

Each call creates a new table. Multiple tables can coexist for the same source (different models, different columns):

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use jammi_ai::session::InferenceSession;
use jammi_db::store::CachePolicy;
async fn ex(session: &InferenceSession) -> jammi_db::error::Result<()> {
session.generate_text_embeddings("patents", "all-MiniLM-L6-v2", &["abstract".into()], "id", CachePolicy::Bypass, None).await?;
session.generate_text_embeddings("patents", "bge-small-en-v1.5", &["title".into()], "id", CachePolicy::Bypass, None).await?;
Ok(()) }
}

When searching, the latest ready embedding table is used by default.

Import precomputed embeddings

When the vectors already exist — computed by an offline batch, migrated from another store, or upserted from a remote encoder — register them directly as a ready embedding table instead of re-running the model. The input is a Parquet object with a _row_id (Utf8) column and a vector (FixedSizeList<Float32> of width dimensions) column, one row per key.

Rust

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use jammi_ai::session::InferenceSession;
async fn ex(session: &InferenceSession) -> jammi_db::error::Result<()> {
use jammi_db::storage::StorageUrl;

let vectors = StorageUrl::parse("file:///data/precomputed.parquet")?;
let record = session.import_embeddings(
    "patents",
    "sentence-transformers/all-MiniLM-L6-v2",
    &vectors,
    "id",
    &["abstract".to_string()],
    384,
).await?;

println!("Imported {} rows, {} dimensions", record.row_count, record.dimensions().unwrap());
Ok(()) }
}

Python

db.import_embeddings(
    source="patents",
    model="sentence-transformers/all-MiniLM-L6-v2",
    vectors_url="file:///data/precomputed.parquet",
    key="id",
    text_columns=["abstract"],
    dimensions=384,
)

The result is indistinguishable from a generated table — same (_row_id, _source_id, _model_id, vector) schema, same sidecar ANN index — so search queries it exactly like any other embedding table. Three behaviours are specific to import:

• Vectors are L2-normalized on import. Every embedding table holds unit vectors (the cosine ANN sidecar assumes it), so each incoming vector is normalized and a zero-norm vector is rejected — it cannot be cosine-searched.
• The model is validated, not loaded. model is parsed to its canonical form and recorded as the table’s derivation provenance; import never loads the encoder or downloads weights, so it needs no GPU. key and text_columns are recorded as catalog provenance (which source column the keys came from, which content columns produced the vectors); the physical key stays _row_id.
• The table is recompute-inert. The engine did not compute these vectors, so a recompute of an imported table is a typed refusal rather than a re-run guessed from its columns.

The input vectors are read fully into memory; a streaming variant is future work.

Supported models

Any encoder model on HuggingFace Hub with safetensors weights. Supported architectures:

BERT family — BERT, RoBERTa, DistilBERT, CamemBERT, XLM-RoBERTa:

• sentence-transformers/all-MiniLM-L6-v2 (384-dim, fast)
• sentence-transformers/all-mpnet-base-v2 (768-dim, higher quality)
• BAAI/bge-small-en-v1.5, BAAI/bge-base-en-v1.5

ModernBERT — modernized encoder with rotary embeddings, 8192-token context, GeGLU:

• answerdotai/ModernBERT-base (768-dim)
• answerdotai/ModernBERT-large (1024-dim)

Or any local directory with config.json + model.safetensors + tokenizer.json. The architecture is detected automatically from model_type in config.json.

Use a local model:

#![allow(unused)]
fn main() {
extern crate jammi_ai;
use jammi_ai::model::ModelSource;
let model = ModelSource::local("/path/to/my-model");
}

Pooling

Pooling — how per-token hidden states collapse into one sentence vector — is model-declared, not hardcoded. On load, the engine reads the model’s 1_Pooling/config.json (the sentence-transformers convention) and pools with the strategy it declares: pooling_mode_cls_token selects CLS pooling (first token — the mode BGE, GTE, and many E5-family models require), and pooling_mode_mean_tokens (or pooling_mode_mean_sqrt_len_tokens, which is exactly equivalent after the mandatory L2 normalization) selects mean pooling. Max and weighted-mean pooling are also supported.

A model whose repository ships no 1_Pooling/ directory — many bare BERT checkpoints — falls back to mean pooling, the historical sentence-transformers default. A model whose 1_Pooling/config.json declares a mode the engine cannot represent (e.g. last-token pooling, or more than one enabled mode at once) fails to load rather than silently pooling incorrectly.

Raw inference (no persistence)

To get embeddings as RecordBatch without writing to disk:

Rust

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use std::sync::Arc;
use jammi_ai::session::InferenceSession;
use jammi_db::store::CachePolicy;
async fn ex(session: &Arc<InferenceSession>) -> jammi_db::error::Result<()> {
use jammi_ai::model::{ModelSource, ModelTask};

let model = ModelSource::hf("sentence-transformers/all-MiniLM-L6-v2");
let (_results, _outcome) = session.infer("patents", &model, ModelTask::TextEmbedding, &["abstract".into()], "id", CachePolicy::Bypass).await?;
Ok(()) }
}

Python

results = db.infer(
    source="patents",
    model="sentence-transformers/all-MiniLM-L6-v2",
    columns=["abstract"],
    task="text_embedding",
    key="id",
)

Each RecordBatch has prefix columns (_row_id, _ordinal, _source, _model, _status, _error, _latency_ms) plus task-specific columns (e.g., vector for embeddings). _ordinal is a stream-scoped, 0-based row counter in model emission order; infer rows read back ordered by _row_id, _ordinal. The materialised embedding table generate_embeddings registers keeps only _row_id, _source_id, _model_id, vector — no _ordinal, since its _row_id is unique by construction.

Error handling

Inference never panics on bad input. _status/_error track per-row input validation, applied before the model ever runs:

The batch continues processing even when individual rows fail this validation. A null key is the one input fault that is never per-row: infer, generate_embeddings and an incremental refresh all refuse the whole call with InvalidKey { column, null_count } before any model call — _row_id is non-nullable by contract, so a silent drop would break the per-row disclosure this table promises. A model-forward failure itself — a broken kernel, a contiguity/PTX/dtype mismatch, or a model incapable of the requested task — is always systemic (every row fails identically), never a per-row event, so it fails the whole infer/embedding call with an error rather than being served as an all-"error" relation or an empty “ready” embedding table.

Dynamic batch sizing

The runner starts with the configured inference.batch_size (default: 32). If an out-of-memory error occurs:

1. Halve the batch size
2. Retry (up to 3 times)
3. If OOM persists at batch size 1, the call fails with an error

The reduced batch size is sticky for the remainder of the stream.

Crash recovery

If the process dies mid-generation, the table is left in “building” status. On the next session start, recovery runs automatically:

• Parquet missing — mark as failed
• Parquet corrupt — delete file, mark as failed
• Parquet valid but stuck in “building” — promote to “ready”, rebuild ANN index

No data is lost if the Parquet file was fully written.

DataFusion integration

Result tables are automatically registered in DataFusion and queryable via SQL:

#![allow(unused)]
fn main() {
extern crate jammi_db;
extern crate jammi_ai;
extern crate tokio;
use jammi_ai::session::InferenceSession;
use jammi_db::catalog::result_repo::ResultTableRecord;
async fn ex(session: &InferenceSession, record: &ResultTableRecord) -> jammi_db::error::Result<()> {
let results = session.sql(&format!(
    "SELECT _row_id, _source_id FROM \"jammi.{}\" LIMIT 10",
    record.table_name
)).await?;
Ok(()) }
}