Module llamacpp 

Remote embedding provider for a llama.cpp llama-server.

llama-server exposes an OpenAI-compatible POST /v1/embeddings endpoint, but it differs from hosted embedding APIs in one way that matters for production ingestion: it never truncates embedding input. Encoder models such as BGE (BERT) are non-causal, so the whole prompt has to fit in a single physical batch (--ubatch-size) and inside the context window (--ctx-size); anything longer is rejected with an HTTP error instead of being cut to size.

This provider therefore runs a deterministic, tokenizer-aware pre-pass before every embedding call:

1. Texts that provably fit are sent as plain strings. Every WordPiece token consumes at least one character, so a text with at most max_input_tokens - 2 characters cannot exceed the budget once the two special tokens ([CLS]/[SEP]) are added.
2. Longer texts are sent to the server’s native POST /tokenize endpoint with add_special: true, which returns the exact id sequence the server would embed, special tokens included.
3. If that sequence exceeds max_input_tokens, the provider keeps the first max_input_tokens - 1 ids and re-appends the trailing special token, then sends the token array (not text) to /v1/embeddings. llama.cpp passes integer arrays through verbatim, so the server embeds exactly the bounded sequence — no lossy detokenize round trip.

Batches preserve input order and produce exactly one vector per input.

Options

key	type	required	meaning
base_url	string	yes	OpenAI-compatible root including /v1, e.g. http://127.0.0.1:8080/v1
tokenizer_base_url	string	no	server root for /tokenize; defaults to base_url with a trailing /v1 removed
max_input_tokens	integer ≥ 3	yes	total token budget including special tokens (512 for BGE)
embedding_dimensions	integer > 0	yes	expected vector width (384 for BGE Small)
api_key_env	string	no	env var holding a bearer token; omit when the server runs without --api-key
request_timeout_secs	integer > 0	no	per-HTTP-request timeout, default 60

max_input_tokens must not exceed the server’s --ubatch-size or --ctx-size; if it does, the server’s rejection surfaces as RuntimeError::InferenceError naming the option.

Structs

LlamaCppEmbeddingModel

Embedding model backed by a llama.cpp server.

RemoteLlamaCppProvider

Remote provider for a llama.cpp llama-server. Supports ModelTask::Embed only.

Constants

DEFAULT_REQUEST_TIMEOUT_SECS

Default per-request HTTP timeout when request_timeout_secs is unset.

MIN_MAX_INPUT_TOKENS

Smallest meaningful max_input_tokens: two special tokens plus one content token.

PROVIDER_ID

Provider id used in ModelAliasSpec::provider_id.