Struct ModelRuntime 

pub struct ModelRuntime { /* private fields */ }

The central runtime that owns registered providers and a catalog of model aliases.

Obtain an instance via ModelRuntime::builder() and the ModelRuntimeBuilder. Once built, use embedding, reranker, or generator to obtain typed, instrumented model handles.

Models are loaded lazily on first access (unless configured for eager or background warmup) and cached in an internal registry so that subsequent requests for the same model are served instantly.

Implementations

impl ModelRuntime

pub fn builder() -> ModelRuntimeBuilder

Create a new ModelRuntimeBuilder for configuring and constructing a runtime.

pub async fn register(&self, spec: ModelAliasSpec) -> Result<()>

Register a new model alias at runtime.

pub async fn contains_alias(&self, alias: &str) -> bool

Check if an alias exists in the catalog.

pub async fn prefetch_all(&self) -> Result<()>

Pre-load and cache every model in the catalog.

Models already loaded are skipped. Fails fast on the first error. Call this during application startup to avoid cold-start latency on first inference.

pub async fn prefetch(&self, aliases: &[&str]) -> Result<()>

Pre-load and cache specific aliases.

Returns an error immediately if an alias is not found in the catalog or if any model fails to load. Models already loaded are skipped.

pub async fn embedding(&self, alias: &str) -> Result<Arc<dyn EmbeddingModel>>

Resolve, load (if necessary), and return an instrumented EmbeddingModel handle for the given alias.

pub async fn reranker(&self, alias: &str) -> Result<Arc<dyn RerankerModel>>

Resolve, load (if necessary), and return an instrumented RerankerModel handle for the given alias.

pub async fn generator(&self, alias: &str) -> Result<Arc<dyn GeneratorModel>>

Resolve, load (if necessary), and return an instrumented GeneratorModel handle for the given alias.