Skip to main content

Module chunker

Module chunker 

Source
Expand description

Content-defined chunker — a thin wrapper over the gearhash crate.

This reproduces the upstream xet-core Chunker (deduplication/src/chunking.rs at tag git-xet-v0.2.0) byte-for-byte: same gearhash::Hasher engine, same TARGET_CHUNK_SIZE (64 KiB), same minimum (target / MINIMUM_CHUNK_DIVISOR = 8 KiB) and maximum (target * MAXIMUM_CHUNK_MULTIPLIER = 128 KiB) bounds, and the same left-shifted FastCDC mask. Chunk hashing delegates to merklehash::compute_data_hash, i.e. BLAKE3 keyed with the XET DATA_KEY, which is exactly what deduplication::Chunk::new does.

Reusing gearhash + merklehash directly (rather than depending on the deduplication crate) avoids pulling the heavy xet_config / xet_runtime / mdb_shard / progress_tracking tree into cas-serve, which is intended to stay lightweight (no libtorch). The boundary logic below is a direct port of xet-core’s, so chunks produced here are identical to those produced by a stock xet-core client.

Structs§

Chunk
A content-defined chunk: its BLAKE3-keyed hash and its bytes.
Chunker
Content-defined chunker using the FastCDC / gear-hash algorithm.

Constants§

MAXIMUM_CHUNK_MULTIPLIER
MAX_XORB_BYTES
XORB limits, matching deduplication/src/constants.rs: a xorb (eXclusive-OR Block) holds at most 64 MiB across at most 8192 chunks.
MAX_XORB_CHUNKS
MINIMUM_CHUNK_DIVISOR
TARGET_CHUNK_SIZE
XET chunking constants, matching deduplication/src/constants.rs.

Functions§

chunk_all
Convenience: chunk a complete byte slice in one shot, flushing the tail.