Skip to main content

Module batched_lora

Module batched_lora 

Source
Expand description

Batched multi-tenant LoRA forward pass

Gathers per-tenant A/B matrices, stacks them into batched tensors, and uses bmm for efficient parallel LoRA computation. This avoids sequential per-tenant forward passes.

Structsยง

BatchedLoRAForward
Batched LoRA forward pass for multi-tenant inference.