Skip to main content

least_loaded_select

Function least_loaded_select 

Source
pub fn least_loaded_select(candidates: &[InferenceServerInfo]) -> Option<usize>
Expand description

Select the least-loaded candidate (lowest active_sessions, ties broken by most free GPU memory). Returns None for an empty set.