Skip to main content

handle_infer

Function handle_infer 

Source
pub async fn handle_infer(
    model_ref_str: &str,
    prompt: &str,
    image_path: Option<String>,
    max_tokens: Option<usize>,
    temperature: Option<f32>,
    top_p: Option<f32>,
    top_k: Option<usize>,
    repeat_penalty: Option<f32>,
    seed: Option<u32>,
    sync: bool,
    signing_key: SigningKey,
) -> Result<()>
Expand description

Handle infer command

Runs inference via InferenceService, which:

  • Enforces authorization via PolicyManager
  • Auto-loads adapters from model directory
  • Training is always DISABLED - this is a read-only inference command

For inference with training (TTT), use hyprstream training infer instead.

ยงParameters

  • signing_key: Ed25519 signing key for request authentication