Skip to main content

BLOCK_SIZE

Constant BLOCK_SIZE 

Source
pub const BLOCK_SIZE: usize = 256; // 256usize
Expand description

Number of tokens per block in the paged KV cache. 256 balances fragmentation overhead vs block reuse granularity.