Skip to main content

KV_BLOCK_SIZE_DEFAULT

Constant KV_BLOCK_SIZE_DEFAULT 

Source
pub const KV_BLOCK_SIZE_DEFAULT: usize = 256;
Expand description

Default tokens-per-block for paged KV storage.

This mirrors crate::runtime::kv_cache::BLOCK_SIZE; the two are asserted equal by a unit test so the const lives here without pulling the tch-backed kv_cache module into this module’s dependency graph.