Module llamacpp 

Remote provider for a llama.cpp llama-server.

llama-server speaks a near-complete OpenAI-compatible API over one loaded model, plus a handful of native endpoints of its own. This module is the provider shell: the option plumbing every task shares, the HTTP seam, and the error classifier. Each task lives in its own sibling module:

task	module	endpoint
ModelTask::Embed	[embed]	POST /v1/embeddings + native /tokenize
ModelTask::Generate	[chat]	POST /v1/chat/completions
ModelTask::Rerank	[rerank]	POST /v1/rerank
ModelTask::DocumentExtract	[docs]	POST /v1/chat/completions

Shared options

key	type	required	meaning
base_url	string	yes	OpenAI-compatible root including /v1, e.g. http://127.0.0.1:8080/v1
api_key_env	string	no	env var holding a bearer token; omit when the server runs without --api-key
request_timeout_secs	integer > 0	no	per-HTTP-request timeout, default 60

Per-task options are documented on the module that reads them.

Input length

llama-server never silently truncates. A prompt that does not fit the physical batch (--ubatch-size) or the context window (--ctx-size) is rejected with an HTTP error, which [map_llamacpp_status] turns into a non-retryable RuntimeError::InferenceError naming the remedy. That is a deterministic configuration problem rather than a server outage, so these rejections are also kept off the circuit breaker — see embed::LlamaCppEmbeddingModel::embed.

Structs

LlamaCppDocumentExtractor

Document extraction backed by a document VLM on a llama.cpp server.

LlamaCppEmbeddingModel

Embedding model backed by a llama.cpp server.

LlamaCppGeneratorModel

Text and vision generation backed by a llama.cpp server.

LlamaCppRerankerModel

Reranker backed by a llama.cpp server.

RemoteLlamaCppProvider

Remote provider for a llama.cpp llama-server.

Constants

DEFAULT_REQUEST_TIMEOUT_SECS

Default per-request HTTP timeout when request_timeout_secs is unset.

MIN_MAX_INPUT_TOKENS

Smallest meaningful max_input_tokens: two special tokens plus one content token.

PROVIDER_ID

Provider id used in ModelAliasSpec::provider_id.