Expand description
Content-defined chunker — a thin wrapper over the gearhash crate.
This reproduces the upstream xet-core Chunker
(deduplication/src/chunking.rs at tag git-xet-v0.2.0) byte-for-byte:
same gearhash::Hasher engine, same TARGET_CHUNK_SIZE (64 KiB), same
minimum (target / MINIMUM_CHUNK_DIVISOR = 8 KiB) and maximum
(target * MAXIMUM_CHUNK_MULTIPLIER = 128 KiB) bounds, and the same
left-shifted FastCDC mask. Chunk hashing delegates to
merklehash::compute_data_hash, i.e. BLAKE3 keyed with the XET DATA_KEY,
which is exactly what deduplication::Chunk::new does.
Reusing gearhash + merklehash directly (rather than depending on the
deduplication crate) avoids pulling the heavy xet_config /
xet_runtime / mdb_shard / progress_tracking tree into cas-serve,
which is intended to stay lightweight (no libtorch). The boundary logic
below is a direct port of xet-core’s, so chunks produced here are
identical to those produced by a stock xet-core client.
Structs§
- Chunk
- A content-defined chunk: its BLAKE3-keyed hash and its bytes.
- Chunker
- Content-defined chunker using the FastCDC / gear-hash algorithm.
Constants§
- MAXIMUM_
CHUNK_ MULTIPLIER - MAX_
XORB_ BYTES - XORB limits, matching
deduplication/src/constants.rs: a xorb (eXclusive-OR Block) holds at most 64 MiB across at most 8192 chunks. - MAX_
XORB_ CHUNKS - MINIMUM_
CHUNK_ DIVISOR - TARGET_
CHUNK_ SIZE - XET chunking constants, matching
deduplication/src/constants.rs.
Functions§
- chunk_
all - Convenience: chunk a complete byte slice in one shot, flushing the tail.