pub struct BatchedLoRAForward { /* private fields */ }Expand description
Batched LoRA forward pass for multi-tenant inference.
Gathers A/B matrices from each tenant’s delta, pads to a common rank, stacks into batch tensors, and performs bmm for efficient computation.
Implementations§
Source§impl BatchedLoRAForward
impl BatchedLoRAForward
Sourcepub fn new(
delta_pool: Arc<DeltaPool>,
max_rank: usize,
scaling: f64,
device: Device,
) -> Self
pub fn new( delta_pool: Arc<DeltaPool>, max_rank: usize, scaling: f64, device: Device, ) -> Self
Create a new batched LoRA forward module.
§Arguments
delta_pool- Shared delta pool containing per-tenant LoRA weightsmax_rank- Maximum LoRA rank to pad to (default: 8)scaling- LoRA scaling factor (alpha / rank)device- Device for tensor allocation
Sourcepub fn batched_lora_forward(
&self,
x: &[&Tensor],
module_name: &str,
tenant_ids: &[Subject],
layer_idx: usize,
) -> Result<Vec<Tensor>>
pub fn batched_lora_forward( &self, x: &[&Tensor], module_name: &str, tenant_ids: &[Subject], layer_idx: usize, ) -> Result<Vec<Tensor>>
Compute batched LoRA corrections for a single module across all provided tenants.
§Arguments
x- Input tensor per tenant: Vec of [1, seq_len, in_features] tensorsmodule_name- Name of the target module (e.g., “q_proj”)tenant_ids- Tenant IDs in the same order asxlayer_idx- Layer index for per-layer delta lookup
§Returns
Vec of LoRA correction tensors, one per tenant. Each is [1, seq_len, out_features]. Returns zero tensor for tenants without a delta.
Sourcepub fn single_tenant_forward(
&self,
x: &Tensor,
module_name: &str,
tenant_id: &Subject,
layer_idx: usize,
) -> Result<Tensor>
pub fn single_tenant_forward( &self, x: &Tensor, module_name: &str, tenant_id: &Subject, layer_idx: usize, ) -> Result<Tensor>
Compute batched LoRA correction for a single tenant (most common case).
Optimized path for batch_size=1 that avoids allocation overhead.
§Arguments
x- Input tensor [1, seq_len, in_features]module_name- Name of the target moduletenant_id- The tenant IDlayer_idx- Layer index for per-layer delta lookup
§Returns
LoRA correction tensor [1, seq_len, out_features], or zero tensor if no delta.
Sourcepub fn delta_pool(&self) -> &Arc<DeltaPool> ⓘ
pub fn delta_pool(&self) -> &Arc<DeltaPool> ⓘ
Get the delta pool reference.
Auto Trait Implementations§
impl Freeze for BatchedLoRAForward
impl !RefUnwindSafe for BatchedLoRAForward
impl Send for BatchedLoRAForward
impl Sync for BatchedLoRAForward
impl Unpin for BatchedLoRAForward
impl UnsafeUnpin for BatchedLoRAForward
impl !UnwindSafe for BatchedLoRAForward
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
§impl<T> ServiceExt for T
impl<T> ServiceExt for T
§fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>where
Self: Sized,
fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>where
Self: Sized,
Apply a transformation to the response body. Read more
§fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>where
Self: Sized,
fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>where
Self: Sized,
High level tracing that classifies responses using HTTP status codes. Read more
§fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>where
Self: Sized,
fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>where
Self: Sized,
High level tracing that classifies responses using gRPC headers. Read more