ModelProvider

uni_xervo::traits

Trait ModelProvider 

Source 
pub trait ModelProvider: Send + Sync {
    // Required methods
    fn provider_id(&self) -> &'static str;
    fn capabilities(&self) -> ProviderCapabilities;
    fn load<'life0, 'life1, 'async_trait>(
        &'life0 self,
        spec: &'life1 ModelAliasSpec,
    ) -> Pin<Box<dyn Future<Output = Result<LoadedModelHandle>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;
    fn health<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = ProviderHealth> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided method
    fn warmup<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait { ... }
}
Expand description

A pluggable backend that knows how to load models for one or more ModelTask types.

Providers are registered with ModelRuntimeBuilder::register_provider and are identified by their provider_id (e.g. "local/candle", "remote/openai").

Required Methods§

Source

fn provider_id(&self) -> &'static str

Unique identifier for this provider (e.g. "local/candle", "remote/openai").

Source

fn capabilities(&self) -> ProviderCapabilities

Return the set of tasks this provider supports.

Source

fn load<'life0, 'life1, 'async_trait>( &'life0 self, spec: &'life1 ModelAliasSpec, ) -> Pin<Box<dyn Future<Output = Result<LoadedModelHandle>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Load (or connect to) a model described by spec and return a type-erased handle.

The returned LoadedModelHandle is expected to contain an Arc<dyn EmbeddingModel>, Arc<dyn RerankerModel>, or Arc<dyn GeneratorModel> depending on the task.

Source

fn health<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = ProviderHealth> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Report the current health of this provider.

Provided Methods§

Source

fn warmup<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Optional one-time warmup hook called during runtime startup.

Use this for provider-wide initialization such as setting up API clients or pre-caching shared resources. The default implementation is a no-op.

Implementors§