Struct ModelRuntime 

pub struct ModelRuntime { /* private fields */ }

The central runtime that owns registered providers and a catalog of model aliases.

Obtain an instance via ModelRuntime::builder() and the ModelRuntimeBuilder. Once built, use embedding, reranker, or generator to obtain typed, instrumented model handles.

Models are loaded lazily on first access (unless configured for eager or background warmup) and cached in an internal registry so that subsequent requests for the same model are served instantly.

Implementations

impl ModelRuntime

pub fn builder() -> ModelRuntimeBuilder

Create a new ModelRuntimeBuilder for configuring and constructing a runtime.

pub async fn register(&self, spec: ModelAliasSpec) -> Result<()>

Register a new model alias at runtime.

pub async fn contains_alias(&self, alias: &str) -> bool

Check if an alias exists in the catalog.

pub async fn prefetch_all(&self) -> Result<()>

Pre-load and cache every model in the catalog.

Models already loaded are skipped. Fails fast on the first error. Call this during application startup to avoid cold-start latency on first inference.

pub async fn prefetch(&self, aliases: &[&str]) -> Result<()>

Pre-load and cache specific aliases.

Returns an error immediately if an alias is not found in the catalog or if any model fails to load. Models already loaded are skipped.

pub async fn embedding(&self, alias: &str) -> Result<Arc<dyn EmbeddingModel>>

Resolve, load (if necessary), and return an instrumented EmbeddingModel handle for the given alias.

The returned handle is cached per alias so that repeated calls skip spec lookup, key hashing, and wrapper allocation.

pub async fn embedder(&self, alias: &str) -> Result<Arc<dyn EmbeddingModel>>

Resolve a dense text EmbeddingModel by alias.

Agent-noun alias for embedding, matching the image_embedder / sparse_embedder / multi_vector_embedder naming.

pub async fn reranker(&self, alias: &str) -> Result<Arc<dyn RerankerModel>>

Resolve, load (if necessary), and return an instrumented RerankerModel handle for the given alias.

The returned handle is cached per alias so that repeated calls skip spec lookup, key hashing, and wrapper allocation.

pub async fn generator(&self, alias: &str) -> Result<Arc<dyn GeneratorModel>>

Resolve, load (if necessary), and return an instrumented GeneratorModel handle for the given alias.

The returned handle is cached per alias so that repeated calls skip spec lookup, key hashing, and wrapper allocation.

pub async fn raw_tensor_model( &self, alias: &str, ) -> Result<Arc<dyn RawTensorModel>>

Resolve, load (if necessary), and return an instrumented RawTensorModel handle for the given alias.

The returned handle is cached per alias so that repeated calls skip spec lookup, key hashing, and wrapper allocation.

pub async fn image_embedder( &self, alias: &str, ) -> Result<Arc<dyn ImageEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented ImageEmbeddingModel handle for the given alias.

Examples

let embedder = runtime.image_embedder("embed/siglip").await?;
let image = ImageInput::Bytes {
    data: std::fs::read("photo.png").unwrap(),
    media_type: "image/png".to_string(),
};
let result = embedder.embed(vec![image]).await?;
println!("dimension: {}", result.vectors[0].len());

Errors

Returns an error if the alias is unknown, the model fails to load, or the provider does not implement image embedding.

pub async fn audio_embedder( &self, alias: &str, ) -> Result<Arc<dyn AudioEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented AudioEmbeddingModel handle for the given alias.

pub async fn multimodal_embedder( &self, alias: &str, ) -> Result<Arc<dyn MultimodalEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented MultimodalEmbeddingModel handle for the given alias.

pub async fn sparse_embedder( &self, alias: &str, ) -> Result<Arc<dyn SparseEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented SparseEmbeddingModel handle for the given alias.

Errors

Returns an error if the alias is unknown, the model fails to load, or the provider does not implement sparse embedding.

pub async fn multi_vector_embedder( &self, alias: &str, ) -> Result<Arc<dyn MultiVectorEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented MultiVectorEmbeddingModel handle for the given alias.

Errors

Returns an error if the alias is unknown, the model fails to load, or the provider does not implement multi-vector embedding.

pub async fn hybrid_embedder( &self, alias: &str, ) -> Result<Arc<dyn HybridEmbeddingModel>>

Resolve, load (if necessary), and return an instrumented HybridEmbeddingModel handle for the given alias.

The handle serves dense, sparse, and multi-vector heads from a single forward pass; select which to materialize with a HeadSet. Only multi-output graphs with a hybrid preset (e.g. BGEM3Hybrid) resolve here — single-head models use the per-task resolvers.

Errors

Returns an error if the alias is unknown, the model has no hybrid preset, the model fails to load, or the loaded handle lacks the hybrid capability.

pub async fn nlp_model(&self, alias: &str) -> Result<Arc<dyn NlpModel>>

Resolve, load (if necessary), and return an instrumented NlpModel handle for the given alias.

pub async fn document_extractor( &self, alias: &str, ) -> Result<Arc<dyn DocumentExtractionModel>>

Resolve, load (if necessary), and return an instrumented DocumentExtractionModel handle for the given alias.

Examples

let extractor = runtime.document_extractor("docext/olmocr").await?;
let page = ImageInput::Bytes {
    data: std::fs::read("page.png").unwrap(),
    media_type: "image/png".to_string(),
};
let options = DocExtractOptions {
    output: DocOutputFormat::Markdown,
    include_tables: true,
    include_formulas: true,
    include_bboxes: false,
};
let pages = extractor.extract(vec![page], options).await?;
println!("{}", pages[0].plain_markdown);

Errors

Returns an error if the alias is unknown, the model fails to load, or the provider does not implement document extraction.

pub async fn transcriber( &self, alias: &str, ) -> Result<Arc<dyn TranscriptionModel>>

Resolve, load (if necessary), and return an instrumented TranscriptionModel handle for the given alias.

pub async fn ocr_model(&self, alias: &str) -> Result<Arc<dyn OcrModel>>

Resolve, load (if necessary), and return an instrumented OcrModel handle for the given alias.

Examples

let ocr = runtime.ocr_model("ocr/ppocr-en").await?;
let image = ImageInput::Bytes {
    data: std::fs::read("scan.png").unwrap(),
    media_type: "image/png".to_string(),
};
let results = ocr.recognize(vec![image]).await?;
println!("{}", results[0].plain_text);

Errors

Returns an error if the alias is unknown, the model fails to load, or the provider does not implement OCR.