Expand description
Batched multi-tenant LoRA forward pass
Gathers per-tenant A/B matrices, stacks them into batched tensors, and uses bmm for efficient parallel LoRA computation. This avoids sequential per-tenant forward passes.
Structsยง
- Batched
LoRA Forward - Batched LoRA forward pass for multi-tenant inference.